WORKING METHOD AND EQUIPMENT AND SYSTEM WITH DOUBLE MOUSE SUPPORT

专利摘要:
A computer-implemented method (5000) for a computer device (4501) with two pointing devices (4503R, 4503L) and / or a mouse filter driver (4563) and / or a hub (5420) configured to send certain messages from the pointing devices. converting to motion sequences, comprising the steps of: b) allowing the operating system (4560) to display the native mouse cursor (705); d) providing a message processor (4552; 4652) to process raw input messages from the pointing devices; e) keeping (5005) a first and second position (Pos1, Pos2) according to movements of the first and second pointing device; f) adjusting (5006) a position of the native mouse cursor (705) according to movements of the first or second pointing device (4503R, 4503L) that was most recently moved or operated; g) detect (5007) and convert the motion sequences into a corresponding command. An overlay application. A mouse filter driver. A special mouse. A special hub. A computer device. A computer system. An operating system with two mouse cursors.
公开号:BE1026980B1
申请号:E20195219
申请日:2019-04-03
公开日:2020-08-24
发明作者:Lambert Jacobs
申请人:Inventrans Bvba；
IPC主号:

专利说明:

Field of the Invention The present invention generally relates to the field of computers having a graphical user interface and multiple pointing devices.
More specifically, the present invention relates to a computer application (e.g., an overlay application), a mouse filter driver, a pointing device (e.g., a mouse), a hub, a computer device, and a dual mouse computer system. support, and on an operating system with two mouse cursors.
Background of the Invention FIG. 1 shows a schematic block diagram of a classical computer system 100 comprising a computer device 101 (e.g., a desktop computer) having a keyboard input connected to a keyboard device 102, and a mouse input connected to a mouse device 103, and a video output connected to a display screen or monitor or screen 104. The computer device 101 of FIG. 1 includes one or more central processing units (CPU) that run an operating system O / S with a graphical user interface GUI, such as, for example, Windows 7 or Windows 10 from Microsoft Corporation, or mac OS 10.12 Sierra or mac OS 10.13 High Sierra from Apple Inc, or Linux.
In the example of FIG. 1, the computer 101 also runs two applications: a PDF document viewer 107a (e.g.
Acrobat Reader) is displayed on the left side of the screen 104, and a text editor 107b (e.g.
Microsoft Word) shown on the right side of the screen 104. Such a system 100 can be used, for example, by a translator or a proofreader to simultaneously view a source document, for example a granted European patent document with English, French and German claims, and a translation. from the conclusions to another language (eg.
Dutch or Spanish or Italian). While the translation itself will typically be performed using a CAT (computer assisted translation) tool, the final proofreading step is preferably performed by comparing the original PDF document and the final text document, or if the final document has also been converted is to a PDF document, by comparing the two PDF documents.
In proofreading (or verifying and / or correcting the translation), the translator or proofreader must keep shifting his focus back and forth between the visual representations of the two documents 1074, 107b, acquiring the current context over and over again in each of these documents, which is time consuming and mentally demanding.
This cognitive burden can be reduced, for example, by physically pointing to the words of the source text, using a finger of the non-dominant hand, and by indicating corresponding words of the target text using the mouse cursor 105, which can be moved on physically moving the screen by the mouse device 103, which is held in the dominant hand, as shown in FIG. 1 for one-handed person. A left-handed person will typically move the mouse with his left hand, and will point at the screen with his right hand.
But this process is not ideal, partly because the arm cannot always rest on the table, especially with larger screen sizes, or with screens mounted above the table.
Furthermore, when a translation error is found and a correction has to be made, both hands are typically brought to the keyboard 102 to type, such that context is lost again and must be reacquired after the correction is made. It would be advantageous if the computer system 100 provided a mechanism that allows a user to easily switch the focus between two reading locations on the screen 104, which locations can be moved easily and intuitively.
FIG. 2 shows another problem. When reading a document, a user typically uses the mouse cursor to point to the words he is currently reading. Typically when the person wants to highlight or underline some words or add comments, the mouse cursor must be moved to the top of the screen, where the user must click to choose a highlight color, or a typewriter tool, or a comment field, etc., but in doing this, the context of where he was reading is lost, and the context must be reacquired, which is time consuming. It would be advantageous if the computer system 200 provided a mechanism that allows a user to easily switch the focus between a reading location and a work location, which locations can be moved easily and intuitively.
FIG. 3 shows another problem. When writing or checking certain texts (e.g., claims of a patent application), it may be desirable to compare two different parts of a single text document, e.g. a first part containing product claims and a second part containing method claims. contains. In practice, the user sometimes wants to change both text portions, which is quite simple for the lower text portion of the example shown in FIG. 3, because that is where the mouse cursor 305 is located, but is not easy for the top portion of text where the finger is located. The user can swap the position of the finger and mouse cursor, but that is not easy in practice without re-acquiring context at both locations, which is time consuming. It would be advantageous if the computer system 300 provided a mechanism that allows a user to easily switch between two different locations on the screen for reading and / or editing, which locations can be moved easily and intuitively.
FIG. 4 shows another problem. When transcribing (transferring) a sound clip (i.e. converting narration into a written document), a user can use an audio player application (e.g. VLC Media Player) to play a sound clip using a start / stop button, and a text editor application for typing the corresponding text information. Using known computers, the user would typically move the mouse cursor to the audio player, click the audio player window 417 to activate the audio player application, click the start button, let the audio player play for a few seconds, click the start / pause button to pause the audio player, move the mouse cursor to the text editor, click on the text editor window to activate the text editor application, optionally click again to reposition the mouse cursor in the text document, type in the corresponding text using the keyboard , move the mouse cursor 405 back to the audio player, and repeat the same steps over and over until the audio clip is completely transcribed. Moving the mouse cursor 405 back and forth between the start / stop button of the audio player 417 and the current position in the text editor 407 is time consuming, especially since the mouse cursor must be positioned accurately in order not to click the wrong button, for example on a rewind button, because an accidental click on this button would take even more time, as the user will have to find the audio clip. It would be advantageous if the computer system 400 provided a mechanism that allows the user to work at two different screen locations using both hands, especially when one of these locations has a substantially fixed position on the screen. Summary of the Invention It is an object of embodiments of the present invention to provide a computer-implemented method, a computer program product, a mouse filter driver, a mouse, a hub, a computing device and a computer system for providing dual mouse support. , as well as an operating system with two mouse cursors.
It is an object of embodiments of the present invention to provide a mechanism to enhance or extend the functionality of a computing device running an operating system (O / S) with a graphical user interface (GUI) to support two pointing devices instead of just one.
It is a specific purpose of such embodiments that the mechanism functions correctly when the first and second mouse devices are operated separately (move, scroll, click).
It is a specific purpose of such embodiments for the mechanism to function correctly when both mouse devices are moved simultaneously.
It is a specific purpose of such embodiments for the mechanism to function correctly when one mouse device is scrolled and the other mouse device is moved at the same time.
It is a specific purpose of such embodiments for the mechanism to function correctly when both mouse devices are scrolled (nearly) simultaneously.
It is a specific purpose of such embodiments for the mechanism to function correctly when one mouse device is clicked simultaneously and the other mouse device is moved.
It is a specific purpose of such embodiments that the mechanism continues to function correctly when one mouse device is clicked or dragged, and the other mouse device is moved or scrolled or clicked.
It is an object of embodiments of the present invention to provide a computer-implemented method to enhance or extend the functionality of an O / S and GUI, e.g., an existing O / S and GUI that contains a single native mouse cursor or pointer. provides and maintains a single native cursor position, in order to keep track of two pointer positions or screen positions, and optionally display two visible objects at these two pointer positions, which objects are individually movable by means of two pointer positions.
It is an object of embodiments of the present invention to provide a computer-implemented method for enhancing or extending the functionality of an O / S and GUI, e.g., an existing O / S and GUI that provides a single native mouse cursor or pointer and maintains a single native cursor position to display at least two visible objects and track at least two object positions, which objects are individually movable by means of at least two pointing devices.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method where one of the visible objects is the native mouse cursor or pointer provided by the O / S and the GUI.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method that responds differently to movements of the at least two pointing devices.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method that responds differently to movements of the at least two pointing devices, and / or to button presses and button releases and button clicks. of one or both of the pointing devices, and / or on scroll wheel movements of one or both of the pointing devices.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method for enhancing the functionality of the Windows operating system of Microsoft Corporation, headquartered in Redmond, Washington, e.g., Windows 7 or Windows 10 or later versions of Windows.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method for enhancing the functionality of Apple Inc.'s macOS. headquartered in Cupertino, California, US, e.g. mac OS Sierra or mac OS High Sierra or later.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method for improving the functionality of other Operating Systems, such as e.g.
Linux.
It is an object of certain embodiments of the present invention to provide such a computer-implemented method, wherein the dual mouse functionality is provided by a graphics overlay application, and preferably without the overlay application running in elevated mode.
It is also an object of embodiments of the present invention to provide a special pointing device (e.g., a special mouse) that allows a user to scroll with two such pointing devices at the same time, or to move one such pointing device and scroll the other at the same time, or moving one such pointing device and clicking simultaneously with the other, eg on a computer running a special overlay application.
It is also an object of embodiments of the present invention to provide a dedicated hub that allows a user to scroll with two classic pointing devices (e.g., two HID-compatible mice) at the same time, or move one such pointing device while simultaneously moving the other. scrolling, or moving one such pointing device and clicking simultaneously with the other, eg on a computer running a special overlay application.
It is also an object of embodiments of the present invention to provide a computer program product in the form of a mouse filter driver that allows a user to scroll with two classic pointing devices (e.g., two HID compatible mice) at the same time, or one moving such pointing device and scrolling the other at the same time, or moving one such pointing device and clicking simultaneously with the other, eg on a computer running a special overlay application.
It is also an object of embodiments of the present invention to provide an operating system that (inherently) provides two visible mouse cursors related to two pointing devices, and wherein the two pointing devices are allowed to be moved and scrolled at the same time, and wherein one pointing devices may be scrolled while the other is moved, and is either clickable or dragged with either pointing device, but not simultaneously.
It is also an object of embodiments of the present invention to provide a computer system comprising a computer device with an overlay application, and at least one of the aforementioned dedicated hub, or the dedicated mouse filter driver, or with two dedicated pointing devices. which allows a user to scroll with two pointing devices at the same time, or move one such pointing device and scroll the other at the same time, or move one such pointing device and click simultaneously with the other.
It is also an object of embodiments of the present invention to provide a computer system further comprising at least one display device and a keyboard.
These and other objects are achieved by a computer-implemented method for providing an overlay application, and a computer program product, and a mouse filter driver, and a computer device, and a computer system having the features of the independent conclusions.
In a first aspect, the present invention provides a computer-implemented method for supporting two pointing devices in a computer device to which a first pointing device and a second pointing device are connected, and running an operating system with a graphical user interface, the operating system providing a native mouse cursor the method comprising the steps of: b) allowing or configuring the operating system with the graphical user interface to provide the native mouse cursor as a visible object; d) providing a message processor to process raw input messages, and configuring the operating system to send raw input messages from the first and second pointing devices to said message processor; e) maintaining and / or adjusting a first position in accordance with movements of the first pointing device, and adjusting a second position in accordance with movements of the second pointing device; f) adjusting a position of the native mouse cursor in accordance with movements of the first or second pointing device that was most recently moved and / or operated; g) detecting whether the raw input messages form or include a motion sequence in a predetermined format, and upon detection of such motion sequence: determining which pointing device caused this motion sequence; and moving the native mouse cursor to the tracked or adjusted position associated with the pointing device that caused the motion sequence; and sending at least one command selected from the group consisting of a button-pressed command, a button-released command, a scroll-up command, and a scroll-down command to the control system according to the detected motion sequence.
It is an advantage of this method that it allows the application executing this method to work with two pointing devices, even on a computer with an O / S that actually only supports one native mouse cursor.
This method can be performed, for example, by an overlay application, but also by a user application that is not an overlay application.
It is an advantage to use (false) motion messages, because this does not require any modifications to the existing operating system, and because, even when sent to the wrong underlying application, they do not cause significant negative effects for most applications (as opposed to unintentional clicks or unwanted scrolling).
It is noted that it is not strictly necessary for the application to show visual objects itself, but it is sufficient that it can jump the position of the native mouse cursor between two positions (analogous to FIG. 10, but without a black arrow).
It is a big advantage that the native mouse cursor is displayed, and eg not a fixed bitmap, because the mouse cursor can take different shapes depending on the position, eg depending on the underlying application.
By "motion sequence according to a predetermined format" (or according to a predetermined structure) can be meant both a predetermined motion sequence with fixed displacement values, as well as a parameterizable motion sequence (e.g. with a number of fixed displacements, with at least one variable displacement, for the passing a parameter), or a motion sequence containing a predetermined sub-sequence. Movement sequences with predetermined values can be used, for example, for a left button pressed message, right button pressed message, left button released message, right button released message, scroll up message, and a scroll down message. Parameterizable motion sequences can be used, for example, to transmit a motion message, or a scroll-up message, or a scroll-down message.
The group or collection of "predetermined motion sequences" may include, for example, one or more of the following motion sequences: a "left button pressed sequence" associated with a left button pressed message and a left button pressed command; a "left button released" sequence "associated with a left button released message and a left button released command; a "right button pressed sequence" associated with a right button pressed message and a right button pressed command; a "right button released sequence" associated with a right button released message and a right button released command, a "scroll up sequence" associated with a scroll up message and a scroll up command; and a "scroll down sequence" associated with a scroll down message and a scroll down command.
In one embodiment, the sequences with the double displacement values are also checked during detection, i.e., for example, eight sequences or twelve sequences.
In one embodiment, the method further comprises the step of: a) providing a graphical overlay window.
This embodiment is directed to the method performed by an overlay application.
It is an advantage of this method that it extends the functionality of a computer with an O / S that actually has only one native mouse cursor to a computer with two pointing devices (e.g. two mice), which the user can move and / or move individually. operate.
It is a great advantage that it can work with existing desktop applications (without any modification), and with existing mouse device drivers (without any modification).
In one embodiment, in step a), a graphics overlay window is provided as a substantially full screen window consisting mainly of fully transparent pixels.
In one embodiment, the method further includes the step of: c) configuring the graphic overlay window in click-through mode.
In one embodiment, step a) provides a graphics overlay window comprising at least one visible object; and wherein step f) further comprises adjusting a position of said visible object at the position other than the position occupied by the native mouse cursor from the first and second positions.
In this embodiment, the user can see at least two movable objects, e.g., the native mouse cursor, and a bitmap with a black arrow (e.g., as shown in FIG. 8 or FIG. 9 or FIG. 10).
In one embodiment, step a) provides a graphics overlay window comprising at least a first visible object and a second visible object; and wherein step f) further comprises adjusting a position of the first visible object based on the first position, and adjusting a position of the second visible object based on the second position.
For example, in this embodiment, the user can see three movable objects: the native mouse cursor, and two bitmaps with a black arrow (e.g., as shown in FIG. 8 or FIG. 9 or FIG. 10). This has the advantage that the two bitmaps are always visible, regardless of where the native mouse cursor is located.
In one embodiment, each of the motion sequences includes a predetermined sequence of two to twenty moves, or a predetermined sequence of three to ten moves.
It is an advantage of a relatively long sequence that the chance that such a sequence would accidentally be caused by the user to move the mouse is nearly impossible, especially when the sequence has one or more back and forth movements or criss-cross movements. (e.g., as shown in FIG. 47).
In one embodiment, each of the predetermined motion sequences includes a predetermined sequence of at least two or at least three displacements that are different from the zero displacement.
In one embodiment, each of the parameterizable motion sequences comprises a series of two to twenty movements, or a series of three to ten movements. In a preferred embodiment, at least two or at least three or at least four of these movements are predetermined movements.
In one embodiment, a cumulative move of the (e.g., predetermined or parameterizable) motion sequences is a zero move.
It is an advantage of such sequence (s) that the scrolling or clicking of a pointing device by the user does not cause shift or drift from the first or second maintained position (assuming that the physical pointing device is not moved). This is especially important for applications such as shown in FIG. 4 (a), where one pointing device is held stationary.
Preferably, a cumulative move of the parameterizable motion sequences (if used) is also a zero move.
In one embodiment, each of the predetermined motion sequences includes as last move a zero move.
This offers the advantage that the overlay application can work more efficiently, because it does not have to check for each incoming move message whether one of the movement sequences occurred.
Preferably, the last move of the parameterizable motion sequences (if used) is also a zero move.
In one embodiment, step e) comprises adjusting the first position and the second position based on the raw input messages.
However, this is not strictly necessary, and the position can also be determined, for example, by requesting the native cursor position from the operating system.
In one embodiment, step e) includes receiving an input message, and if the received input message is related to the first pointing device, updating the first position; and if the received input message is related to the second pointing device, updating the second position; and step g) comprises: detecting whether a motion sequence with the predetermined structure (e.g., a predetermined motion sequence) exists in the partial stream of received input messages from each pointing device individually.
In one embodiment, step e) comprises: receiving an input message, and if the input message is a motion message, obtaining displacement values (dx, dy) related to the motion message.
In one embodiment, the graphics overlay window is configured as a substantially full screen window consisting essentially (e.g., at least 90%) of fully transparent pixels.
In this embodiment, the overlay window does not need to be explicitly configured in click-through mode.
In one embodiment, the graphic overlay window is configured as a semi-transparent window having an alpha blend value a in the range of 1% to 99%, or in other words, with an alpha transparency value (T) in the range of 1% to 99 %, or from 1% to 90%, or from 1% to 60%, or from 1% to 50%. The lower the alpha transparency value (T), usually the more visible or more distinguishable the first and second visible object (e.g. an arrow or a hand) from the underlying applications.
In one embodiment, the first and second pointing devices are HID compatible pointing devices.
In one embodiment, the operating system is Windows 7 or Windows 10 or a later Windows version from Microsoft Corporation. In one embodiment, the operating system is mac OS Sierra or mac OS High Sierra or a later mac OS version from Apple inc.
In one embodiment, the operating system is Linux.
In a second aspect, the present invention also provides a computer program product containing executable instructions that perform a method according to the first aspect when executed by a computer device to which a first and second pointing device are connected.
In a third aspect, the present invention also provides a computer-implemented method for filtering messages from at least two pointing devices, the method comprising the steps of: a) receiving a message from one of the pointing devices; b) checking whether the received message is a motion message, and if so, sending the motion message [eg. to the operating system or to a port of the computer]; c) checking whether the received message is a button-pressed message, and if so, sending a first (e.g. predetermined) motion sequence; d) checking if the received message is a button released message, and if so, sending a second (e.g. predetermined) motion sequence.
This method can be performed, for example, in a mouse filter driver in the computer device (see FIG. 45 and FIG. 46), or may be performed, for example, in a hub (see FIG. 54). In the case of a mouse filter driver, the individual motion messages or motion sequences are sent, eg by the mouse filter driver, to the operating system. In the case of a hub, the individual motion messages or motion sequences are eg sent from the hub to a port of the computing device, eg an interface or a USB port.
In one embodiment, the method further comprises the steps of: e) checking whether the received message is a scroll-up message, and if so, sending a third (e.g. predetermined or parameterizable) motion sequence; and f) checking whether the received message is a scroll-down message, and if so, sending a fourth (e.g. predetermined or parameterizable) motion sequence.
In one embodiment, step a) further comprises determining from which pointing device the message originated and determining a time of receipt of that message; and the method further comprises a step of checking whether only one pointing device has been used for at least a predetermined period (AT), and if so, sending the message without conversion to a motion sequence.
It is noted that step b) and step g) can also be swapped.
In a fourth aspect, the present invention also provides a computer program product containing executable instructions that perform a method according to the third aspect when executed by a computer device to which a first and a second pointing device are connected, and which includes a graphical user interface that provides a native mouse cursor.
This computer program product can be, for example, a "mouse filter driver".
In a fifth aspect, the present invention also provides a pointing device comprising: a motion detecting mechanism for detecting a two-dimensional motion; at least one button depressible and releasable by a user; a button detecting mechanism for detecting whether the at least one button is pressed or released; a control unit connected to the motion detecting mechanism, and provided for sending a motion message when a motion is detected; characterized in that the control unit is further adapted to transmit a first (e.g. predetermined) motion sequence when it is detected that the at least one button is pressed; and for transmitting a second (e.g., predetermined) motion sequence when the at least one button is detected to be released.
This embodiment is directed to a "special mouse" as shown in FIG. 53.
In one embodiment, the pointing device further includes at least one scroll wheel that is rotatable by a user; and a scroll wheel detecting mechanism for detecting whether the at least one scroll wheel has rotated; and the control unit is connected to the scroll wheel detecting mechanism; characterized in that the controller is further configured to transmit a third (e.g., predetermined or parameterizable) motion sequence when it is detected that the scroll wheel is being turned up; and wherein the controller is further configured to transmit a fourth (e.g., predetermined or parameterizable) motion sequence when it detects that the scroll wheel is being turned down.
In one embodiment, each of the (e.g., predetermined) motion sequences consists of a predetermined sequence of two to twenty moves, or a predetermined sequence of three to ten moves.
In one embodiment, each of the (e.g. parameterizable) motion sequences comprises a series of two to twenty moves, or a series of three to ten movements. In a preferred embodiment, at least two or at least three or at least four of these movements are predetermined movements.
Preferably, the displacements of the (e.g. predetermined) motion sequences are displacements where the X displacement and the Y displacement are values in the range of -3 to +3, or are values in the range of -5 to +5, or values are in the range of -10 to +10.
In one embodiment, a cumulative move of the (e.g., predetermined) motion sequences is a zero move.
Preferably, a cumulative move of the parameterizable motion sequences (if used) is also a zero move.
In one embodiment, each of the (e.g., predetermined) motion sequences has as last displacement a zero displacement (with dx = 0 and dy = 0).
Preferably, the last move of the (e.g., parameterizable) motion sequences (if used) is also a zero move.
In a sixth aspect, the present invention also provides a hub comprising: a first interface for communicating with a first pointing device, a second interface for communicating with a second pointing device, and a third interface for communicating with a computer device; and a controller connected to the first interface for receiving first messages from the first pointing device, and connected to the second interface for receiving second messages from the second pointing device, and connected to the third interface for sending messages to the computer device. ; characterized in that the control unit is adapted to perform a method according to the third aspect.
Thus, this hub performs a method as shown in FIG. 51 or FIG. 52. According to the method of FIG. 51, all button pressed / released messages and all scroll up / down messages are converted to an associated (e.g. predetermined) movement sequence. In the case of FIG. 52a and FIG. 52b, button messages and scroll messages are converted to a motion sequence only when two mice are used.
In one embodiment, each of the predetermined motion sequences consists of a predetermined sequence of two to twenty displacements, or a predetermined sequence of three to ten displacements.
In one embodiment, each of the parameterizable motion sequences comprises a series of two to twenty movements, or a series of three to ten movements. In a preferred embodiment, at least two or at least three or at least four of these movements are predetermined movements.
Preferably, the displacements of the (e.g. predetermined) motion sequences are displacements where the X displacement and the Y displacement are values in the range of -3 to +3, or are values in the range of -5 to +5, or values are in the range of -10 to +10.
In one embodiment, a cumulative move of the (e.g., predetermined) motion sequences is a zero move.
Preferably, a cumulative move of the (e.g., parameterizable) motion sequences (if used) is also a zero move.
In one embodiment, each of the predetermined motion sequences has as last displacement a zero displacement (with dx = 0 and dy = 0). Preferably, the last move of the (e.g., parameterizable) motion sequences (if used) is also a zero move.
In one embodiment, the predetermined period (AT) is a period from 20 ms to 1000 ms, preferably from 20 to 500 ms, or from 25 to 400 ms, or from 50 to 250 ms, e.g., approximately equal to 100 ms , or approximately equal to 150 ms, or approximately equal to 200 ms.
In a seventh aspect, the present invention also provides a computer device having an operating system with a graphical user interface that provides a (e.g. only one) native mouse cursor; and with a computer program product according to the second aspect.
In one embodiment, the computer device further comprises a computer program product according to the fourth aspect.
In an eighth aspect, the present invention also provides a computer system, comprising: a computer device according to the seventh aspect, and a first and a second pointing device operably connected to the computer device.
In one embodiment, the first and second pointing devices are pointing devices according to the fifth aspect.
In one embodiment, the computing device of this computer system includes at least a first USB port for direct connection to the first pointing device, and a second USB port for direct connection to the second pointing device.
In one embodiment, the computer system further includes a hub according to the sixth aspect operably connected between the computing device and the first and second pointing devices.
Preferably, the computer system further includes one or two display devices.
Preferably, the computer system further comprises a keyboard.
In a ninth aspect, the present invention also provides a computer-implemented method executable in a computer device associated with at least a first and a second pointing device, the method comprising the steps of: b) providing a first and a second visible mouse cursor, and tracking of a first and a second position (e.g. pointer position); repeatedly performing: c) receiving an input message from the first and / or the second pointing device; d) updating the first and second position (e.g., pointer position) in accordance with movements of the first and second pointer device; e) adjusting the position of the first and second mouse cursor based on which pointing device was last moved or operated; f) further processing the input message, e.g. by sending or not sending a corresponding command or event to a visual component or to an application window located under the relevant mouse cursor.
This method can be performed, for example, in an operating system that displays two mouse cursors (e.g., an active and a passive one), to support two pointing devices.
In one embodiment, step e) comprises at least one of the following modes: e1) if the first pointing device was last moved or operated, positioning the first mouse cursor at the first pointing position and positioning the second mouse cursor at the second pointing position; and if the second pointing device was last moved or operated, positioning the first mouse cursor at the second pointing position and positioning the second mouse cursor at the first pointing position; e2) positioning the first mouse cursor at the first pointing position, and positioning the second mouse cursor at the second pointing position; e3) positioning the first mouse cursor at the second pointing position; and positioning the second mouse cursor at the first pointing position.
The mode of steps el), e2) and e3) are called the "two-handed mode", the "right-handed mode" and the "left-handed mode", respectively. The operating system may provide the capability (e.g., via the taskbar) to allow a user to quickly and easily switch between these modes, e.g., in a manner as shown in FIG. 31 (a), by allowing the user to choose the dominant hand.
In one embodiment, step f) includes the following: g) checking whether the received message is a button press message, and if so, sending a button press command or button press event from the position of the first mouse cursor to an underlying component or application; h) determining whether the received message is a button release message, and if so, sending a button release command or button release event from the position of the first mouse cursor to an underlying component or application; i) checking whether the received message is a scroll-up message, and if so, sending a scroll-up command or scroll-up event from the position of the first mouse cursor to an underlying component or application; j) checking whether the received message is a scroll-down message, and if so, sending a scroll-down command or scroll-down event from the position of the first mouse cursor to an underlying component or application.
In one embodiment, step g) and h) further comprises updating one or more state variables to indicate whether one or more buttons of the first and / or second pointing device are pressed; and in step f) it is first checked whether a button of the first or second pointing device has been pressed on the basis of these state variables, and it is checked whether the message originates from the pointing device whose button has been pressed; and thereafter, if a result of this test is true, the message is further processed according to steps g) to j); and if a result of this test is false, the message is ignored (in the sense that no command or event is sent to an underlying component or application), and optionally an error message is displayed.
In one embodiment, step g) and h) further comprises updating one or more state variables to indicate whether one or more buttons of the first and / or second pointing device are pressed; and in step f) it is first checked whether a button of the first or the second pointing device has been pressed on the basis of these state variables, and it is checked whether the message is a button released message, and whether the message originates from the pointing device whose button has been pressed; and thereafter, if a result of this test is true, the message is further processed according to steps g) to j); and if a result of this test is false, the message is ignored (in the sense that no command or event is sent to an underlying component or application), and optionally an error message is displayed.
In one embodiment, step d) further comprises determining a first (e.g. visual) component and / or a first application window located at the first pointer position, and determining a second (e.g. visual) component and / or a second application window located at the second pointer position; and in step f) the first and the second component and / or the first and the second application windows are taken into account in the processing of the input message. A variant looks at which component or application is below the cursor positions instead of the cursor positions.
In one embodiment, in step f), if the second pointer position is over the same component and / or over the same application window as the first pointer position, (or in other words, if the first pointer position and the second pointer position are above the same component and / or or the same application window), an error message is displayed in at least one of the following cases: * upon receipt of a button-pressed message from the first pointing device if a button of the second pointing device has already been pressed; * upon receipt of a button-pressed message from the second pointing device if a button of the first pointing device has already been pressed; * upon receipt of a scroll-up message or a scroll-down message from the first pointing device if a button of the second pointing device has already been pressed; * upon receipt of a scroll-up message or a scroll-down message from the second pointing device if a button of the first pointing device has already been pressed.
In one embodiment, in step f), if the first and second mouse cursor are over different (e.g. visual) components or application windows, * upon receipt of a scroll-up message from the first pointing device, a scroll-up message is displayed. command or a scroll-up event sent to the underlying component or application, even if a button of the second pointing device has already been pressed; and * upon receipt of a scroll-down message from the second pointing device, a scroll-down command or a scroll-down event is sent to the underlying component or application, even if a button on the first pointing device is already pressed .
In a tenth aspect, the present invention also includes a computer program product containing executable instructions that implement a method according to the ninth aspect when executed by at least one processor of a computer device to which a first and a second pointing device are connected, e.g. through a hub.
For example, the computer program product may be stored in a local storage medium (e.g., a CD-ROM, or a DVD-ROM, or a flash drive or a hard disk or a USB stick), or may be stored and downloaded via a network.
In an eleventh aspect, the present invention also includes a computer device, comprising: a computer program product according to the tenth aspect.
In a twelfth aspect, the present invention also includes a computing device, comprising: at least one processor performing the method according to the ninth aspect.
In a thirteenth aspect, the present invention also includes a computer system, comprising: a computer device according to the eleventh or twelfth aspect; and a first and a second pointing device operably connected to the computing device.
In a fourteenth aspect, the present invention also includes a computer system according to the thirteenth aspect, wherein the first and second pointing devices are HID compatible mouse devices.
Specific and preferred aspects of the present invention are set out in the appended independent and dependent claims. Features of the dependent claims may be combined with features of the independent claims and with features of other dependent claims as indicated and not just as expressly set forth in the claims.
These and other aspects of the invention will become apparent from and elucidated with reference to the embodiment (s) described below.
Brief Description of the Figures FIG. 1 shows an example of how a translator or proofreader can compare an original document and a translation thereof, using a classic computer system.
FIG. 2 shows an example of how a person reading and editing an electronic text document or editing it or commenting on it can use his finger to "remember" the position where he is reading when he moves the mouse cursor. move to the menu bar to choose a tool or a color, or the like, using a classic computer system.
FIG. 3 shows an example of how a person who is reading or editing a single text document can use their finger to point to one position in the document, and use the mouse cursor to point to another position within that one document, using a classic computer system.
FIG. 4 shows an example of how a person can transcribe an audio file using a classic computer system. FIG. 4 (a) shows how an audio player can be started to play an audio clip. FIG. 4 (b) shows how the audio player is temporarily stopped, and text is being typed into a text document.
FIG. 5 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and to a second pointing device, and outputs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointer position in accordance with movements of the first pointing device (e.g., right mouse), and maintains a second pointing position in accordance with movements of the second pointing device (e.g., left mouse), and adjusts the position of the native mouse pointer / mouse cursor, eg overwrites it, according to movements of the first pointing device (eg right mouse). In addition, the overlay application provides a second visible object (e.g., a line or rectangle) that moves in accordance with movements of the second pointing device (e.g., left mouse). FIG. 5 (a) shows movements of the first visible object in accordance with movements of the first pointing device (e.g., right mouse). FIG. 5 (b) shows movements of the second visible object in accordance with movements of the second pointing device (e.g., left mouse). FIG. 5 {c} is a schematic representation of a so-called "Z-order" of windows or image planes and a "cursor plane" as may be used in the computer system of FIG. 5 (a). FIG. 5 {d} is a high-level flow diagram illustrating a computer-implemented method according to an embodiment of the present invention, such as may be used in the computer system of FIG. 5 (a). FIG. 5 (e) is a more detailed flow diagram illustrating a computer-implemented method according to an embodiment of the present invention, such as may be used in the computer system of FIG. 5 (a). FIG. 5 (f) shows a simplified high level block diagram of hardware and software components located in the computer system of FIG. 5 (a). FIG. 5 (g) shows some lines of pseudo code illustrating part of a simplified message processor as can be used by the overlay application. FIG. 5 (h) shows some lines of pseudo-code illustrating part of a simplified object repositioning routine as can be used by the overlay application to reposition the second visible object.
FIG. 6 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and to a second pointing device, and outputs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay-
application maintains a first pointer position in accordance with movements of the first pointing device (e.g. right mouse), and maintains a second pointer position in accordance with movements of the second pointing device (e.g. left mouse), and adjusts the position of the native mouse pointer / mouse cursor, e.g. overwrites it, in accordance with movements of the first pointing device.
In addition, the overlay application provides a second visible object (e.g., a line or rectangle) that moves in accordance with movements of the second pointing device, and provides a third visible object that moves in accordance with movements of the first pointing device.
FIG. 6 {a} shows movements of the first and third visible object in accordance with movements of the first pointing device (eg right mouse). FIG. 6 (b) shows movements of the second visible object in accordance with movements of the second pointing device (e.g., left mouse). FIG. 6 {c} shows some lines of pseudo-code illustrating part of a simplified position update routine, as can be used by the overlay application to reposition the second and third visible objects.
FIG. 7 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and to a second pointing device, and outputs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention.
The overlay application maintains a first pointer position in accordance with movements of the first pointing device (e.g., right mouse), and maintains a second pointing position in accordance with movements of the second pointing device (e.g., left mouse), and adjusts the position of the native mouse pointer / cursor, e.g. overwrites it, in accordance with movements of the pointing device that was most recently moved.
In addition, the overlay application provides a second visible object (e.g., a line or rectangle) that moves in accordance with movements of the second pointing device (e.g., left mouse), and provides a third visible object that moves in accordance with movements of the first pointing device (eg right mouse). FIG. 7 (a) shows movements of the first and third visible objects in accordance with movements of the first pointing device (e.g., right mouse). FIG. 7 (b) shows movements of the first and second visible object in accordance with movements of the second pointing device (e.g., left mouse). FIG. 7 (c) is a high-level flow chart illustrating a computer-implemented method according to an embodiment of the present invention, such as may be used in the computer system of FIG. 7 (a). FIG. 7 (d) is a more detailed flow diagram illustrating a computer-implemented method according to an embodiment of the present invention, such as may be used in the computer system of FIG. 7 (a). FIG. 7 (e) shows some lines of pseudo code illustrating part of a simplified message processor as can be used by the overlay application.
FIG. 8 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and to a second pointing device, and outputs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointer position in accordance with movements of the first pointing device (e.g., right mouse), and maintains a second pointing position in accordance with movements of the second pointing device (e.g., left mouse), and adjusts the position of the native mouse pointer / mouse cursor, eg overwrites it, according to movements of the first pointing device (eg right mouse). In addition, the overlay application provides a second visible object (e.g., a black arrow) that moves in accordance with movements of the second pointing device (e.g., left mouse). FIG. 8 (a) shows movements of the first visible object in accordance with movements of the first pointing device (e.g., right mouse). FIG. 8 {b} shows movements of the second visible object in accordance with movements of the second pointing device (eg left mouse). FIG. 8 (c) is a schematic representation of a so-called "Z-order" of windows or image planes and a "cursor plane" as may be used in the computer system of FIG. 8 (a).
FIG. 9 shows a computer system according to an embodiment of the present invention, and is a variant of the system shown in FIG. 8. Where the embodiment of FIG. 8 is particularly useful for right-handed people, the embodiment of FIG. 9 especially useful for left-handed people. FIG. 9 (a) shows movements of the second visible object (e.g. a black arrow} in accordance with movements of the first pointing device (e.g. right mouse). FIG. 9 (b) shows movements of the first visible object (e.g. the native mouse cursor / pointer, here in the form of a white arrow) in accordance with movements of the second pointing device (e.g. left mouse) FIG. 9 {c) shows some lines of pseudo-code illustrating part of a simplified message processor as used are by the overlay application running on the computing device 901.
FIG. 10 shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and to a second pointing device, and outputs an O / S and GUI that provides a native mouse pointer / cursor (e.g., a white arrow!) , and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first pointer position in accordance with movements of the first pointing device (e.g., right mouse), and maintains a second pointer position in accordance with movements of the second pointing device (e.g., left mouse), and adjusts the position of the native mouse pointer / cursor, e.g. overwrites it, in accordance with movements of the pointing device that was most recently moved. In addition, the overlay application provides a second visible object (e.g., a black arrow) that occupies the other serviced position. FIG. 10 (a) shows movements of the first visible object (e.g., the native mouse cursor / pointer) in accordance with movements of the first pointing device (the pointing device being moved), while the second visible object is in the other maintained position. FIG. 10 ({b) shows movements of the first visible object in accordance with movements of the second pointing device (the pointing device being moved), while the second visible object (the black arrow) is in the other maintained position.
FIG. 11 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with a special overlay application according to an embodiment of the present invention, and two classic pointing devices directly connected with the computer device.
FIG. 12 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with a special overlay application according to an embodiment of the present invention, and two classic pointing devices connected to the computer equipment via a classic hub.
FIG. 13 shows some lines of pseudo code illustrating part of a first message processing routine as may be used in overlay applications according to embodiments of the present invention.
FIG. 14 (a) to FIG. 14 (c) and FIG. 15 show exemplary sequences of signals or messages from two classic mouse devices, showing how these signals are handled satisfactorily by the first message processing process shown in FIG. 13.
FIG. 16 and FIG. 17 show two exemplary sequences of signals or messages from two classic mouse devices, showing how these signals are handled unsatisfactorily by the message processing process shown in FIG. 13.
FIG. 18 (a) shows some lines of pseudo code illustrating part of a second message processing routine as may be used in overlay applications according to embodiments of the present invention.
FIG. 18 (b) shows a variation of the routine of FIG. 18 (a), as can be used in overlay applications according to embodiments of the present invention.
FIG. 18 (c) shows a more complete version of the routine of FIG. 18 (a), as can be used in overlay applications according to embodiments of the present invention.
FIG. 19 and FIG. 20 show two exemplary sequences of signals or messages from two classic mouse devices, showing how these signals are handled satisfactorily by the second message processing process shown in FIG. 18 (a).
FIG. 21 shows an exemplary sequence of signals or messages from two classic mouse devices, and shows how these signals are handled unsatisfactorily by the second message processing process shown in FIG. 18 (a).
FIG. 22 (a) and FIG. 22 (b) show two exemplary sequences of signals from two special pointing devices according to an embodiment of the present invention, and show how these signals or messages are handled satisfactorily by the second message processing process shown in FIG. 18 (a).
FIG. 23 (a) shows a simplified high-level block diagram of hardware and software components contained in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with a special overlay application according to an embodiment of the present invention, and two special mouse devices according to an embodiment of the present invention.
FIG. 23 (b) shows an exemplary block diagram of a special mouse device according to an embodiment of the present invention, such as may be used in the system of FIG. 23 (a).
FIG. 24 shows a simplified high-level block diagram of hardware and software components contained in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with a special overlay application according to an embodiment of the present invention, and two special mouse devices. according to an embodiment of the present invention, connected to the computing device through a classic hub.
FIG. 25 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with a special overlay application according to an embodiment of the present invention, and comprising two classic mouse devices connected with the computing device via a dedicated adapter or hub or dongle according to an embodiment of the present invention.
FIG. 26 (a) to FIG. 26 (d) show exemplary sequences of signals or messages from the two classic pointing devices of FIG. 25, showing how these signals or messages are handled by a message processing process shown in FIG. 18, after passing through the dedicated adapter or hub or dongle.
FIG. 27 (a) shows an exemplary block diagram of a hub (or plug or adapter) according to an embodiment of the present invention, such as may be used in the system of FIG. 25, which has at least two physical connectors for connection to two classic wired pointing devices.
FIG. 27 (b) shows an exemplary block diagram of a wireless hub (or dongle or adapter) according to an embodiment of the present invention, such as may be used in the system of FIG. 25, which has one or two RF transceivers for operable connection to two classic wireless pointing devices.
FIG. 28 (a) shows an exemplary user interface window, referred to herein as the "mini user interface", which supports a plurality of input modes and a plurality of display modes.
FIG. 28 (b} shows an exemplary popup menu as may be used in embodiments of the present invention.
FIG. 29 shows an exemplary user interface window, referred to herein as the "main user interface", as may be used in embodiments of the present invention, e.g., to select an input mode and / or a display mode, and / or a size and / or to choose a color of the second or further visible object, and / or to choose or adjust a semi-transparency level, and / or to choose a semi-transparent textured background.
FIG. 30 and FIG. 31 (a) show an enlarged view of a portion of FIG. 29.
FIG. 31 (b) is a variation of the portion of FIG. 31 (a), where a feature has been added to force cursor movements to be only horizontal (East / West) or vertical (North / South) or diagonal (North East, North West, South East, South West) .
FIG. 31 (c) shows some lines of pseudo code that can be used in the position update routine to achieve this effect.
FIG. 31 (d) and FIG. 31 (e) illustrate a variation of the displacement transformation routine in which near-horizontal motions are transformed into purely horizontal motions.
FIG. 32 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and two applications (e.g., a PDF document viewer and a word processor), and executes an overlay application according to an embodiment of the present invention . The overlay application is configured to reposition the native mouse cursor and to display two movable lines that move in accordance with movements of the pointing devices.
FIG. 33 shows a variation of the computer system of FIG. 32, wherein the overlay application is configured to reposition the native mouse cursor and display two movable squares that move in accordance with movements of the pointing devices.
FIG. 34 shows another variation of the computer system of FIG. 32, wherein the overlay application is configured to reposition the native mouse cursor and display two visible objects, one object containing a left hand image, and the other object containing a right hand image, which images move in accordance with movements of the indicating devices.
FIG. 35 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and an exemplary application (e.g., a PDF document editor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and display a movable line, in accordance with movements of the pointing devices.
FIG. 36 shows a variation of the computer system of FIG. 35, wherein the overlay application is configured to reposition the native mouse cursor and display a movable cross, in accordance with movements of the pointing devices.
FIG. 37 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and an exemplary application (e.g., a text editor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and display two movable lines in accordance with movements of the pointing devices.
FIG. 38 shows a variation of the computer system of FIG. 37, wherein the overlay application is configured to reposition the native mouse cursor and display two movable crosses, in accordance with movements of the pointing devices.
FIG. 39 shows a computer system according to an embodiment of the present invention, wherein the computer device executes an O / S and GUI and an exemplary application (e.g., text editor), and executes an overlay application according to an embodiment of the present invention. The overlay application is configured to reposition the native mouse cursor and show a vertical line to split the screen into a left part and a right part, and to show two horizontal lines movable vertically, in accordance with movements of the indicating devices.
FIG. 40 shows a variation of the computer system of FIG. 39, the overlay application additionally providing two squares or rectangles within the horizontal lines, which squares are also movable horizontally, in accordance with movements of the pointing devices.
FIG. 41 shows a computer system according to an embodiment of the present invention comprising a special monitor or display screen according to an embodiment of the present invention, directly connectable to a pointing device, the special display screen or monitor adapted to overlay the image provided by the computer device with at least one visible object that is movable in accordance with movements of the pointing device associated with the display screen. The O / S and GUI of the computer device shows the native mouse cursor / pointer, which is movable by the pointing device connected to the computer.
FIG. 42 shows an exemplary block diagram of a display device used in the system of FIG. 41.
FIG. 43 shows a computer system according to an embodiment of the present invention comprising a video adapter device or video interface device according to an embodiment of the present invention, having an input port for receiving first graphic image data from a computer device, the computer device being connected with a first pointing device. The video adapter further includes an output port for providing second graphic image data to a monitor or display screen, the video adapter further including a wired or wireless interface to a second pointing device, the video adapter being adapted to generate at least one visible object, and for overlay the first graphic image data with the at least one visible object, and for moving the visible object in accordance with movements of the second pointing device connected to the video adapter.
FIG. 44 shows an exemplary hardware block diagram of the video adapter device of FIG. 43.
FIG. 45 shows a simplified high-level block diagram of hardware and software components contained in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with an overlay application according to an embodiment of the present invention, and with a mouse filter driver according to an embodiment of the present invention. The computer system further includes two classic mouse devices that are directly connected to the computer device.
FIG. 46 shows a simplified high-level block diagram of hardware and software components contained in a computer system according to an embodiment of the present invention, the computer system comprising a computer device with an overlay application according to an embodiment of the present invention, and with a mouse filter driver according to an embodiment of the present invention. The computer system further includes two classic mouse devices and a classic hub, which mouse devices are connected to the computer device through the hub.
FIG. 47 (a) to FIG. 47 (i) show some exemplary motion sequences that can be used in embodiments of the present invention, e.g., in the computer system of FIG. 45 and FIG. 46.
FIG. 48 shows an exemplary implementation, in pseudo-code, of a highly simplified example, how the overlay application, upon detection of a scroll-up sequence, caused by a scroll-up message from the left or right pointing device, the native mouse cursor to the left or right position, and then send a scroll command to the O / S.
FIG. 49 shows, by means of an example, how the solution works with the motion sequences.
FIG. 50 shows a flow chart of an exemplary computer-implemented method that can be performed by the overlay application used in the system of FIG. 45 or FIG. 46.
FIG. 51 shows a flow chart of an exemplary computer-implemented method that can be performed by the mouse filter driver used in the system of FIG. 45 or FIG. 46.
FIG. 52 (a) shows a flow chart of another exemplary computer-implemented method that can be performed by the mouse filter driver of FIG. 45 or FIG. 46, which can be seen as a variation of the method of FIG. 51.
FIG. 52 {b} shows a flow diagram of a variation of the method of FIG. 51.
FIG. 52 (c) and FIG. 52 (d) show by means of a first and a second example how a mouse filter driver (MFD) that uses the method of FIG. 52 (a) or FIG. 52 (b), processes messages from two pointing devices.
FIG. 52 (e) shows a variation of the method of FIG. 52 (b).
FIG. 53 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, which can be viewed as a variation of the block diagram of FIG. 45, without a mouse filter driver, but with two special mouse devices according to embodiments of the present invention, which are directly connected to the computer device.
FIG. 54 shows a simplified high-level block diagram of hardware and software components located in a computer system according to an embodiment of the present invention, which can be viewed as a variation of the block diagram of FIG. 46, without a mouse filter driver, but with a special hub according to an embodiment of the present invention, which connects the computing device with two classic pointing devices.
FIG. 55 (a) to FIG. 57 illustrate a computer system with an operating system that inherently supports two mouse devices, and that displays two mouse cursors, including a first mouse cursor (also referred to herein as "active mouse cursor"}, and a second mouse cursor (also referred to herein as "passive mouse cursor").
FIG. 55 (a) shows a computer system according to an embodiment of the present invention, wherein a computer device is connected to a first and to a second pointing device, and outputs an O / S and GUI that provides two mouse cursors that track movements of the respective pointing devices, and wherein the O / S and GUI supports separate or nearly simultaneous scrolling movements of the two pointing devices, and permits clicking and dragging with each of the pointing devices, but only one at a time. In this embodiment, the behavior of the two mouse cursors is provided by the operating system itself (or by certain modules thereof), without requiring an overlay application and / or a dedicated mouse device driver and / or a dedicated hub and / or a dedicated pointing device. is required.
FIG. 55 (a) shows movements of the first mouse cursor (active mouse cursor) in accordance with movements of the first pointing device (e.g., right mouse). The second mouse cursor (passive mouse cursor) is now in the second position (also called mouse position).
FIG. 55 {b} shows movements of the first mouse cursor (active mouse cursor) in accordance with movements of the second pointing device (eg, left mouse). The second mouse cursor (passive mouse cursor) is now in the first position.
FIG. 55 (c) is a schematic representation of a so-called "Z-order" of windows or image planes and a "cursor plane" as may be used in the computer system of FIG. 55 (a).
FIG. 55 {d} shows a simplified high-level block diagram of hardware and software components located in the computer system of FIG. 55 (a), wherein the computer system comprises a computer device with a special operating system according to an embodiment of the present invention, which (inherently) supports two classic pointing devices.
FIG. 56 shows a high-level flow diagram of an exemplary computer-implemented method that can be performed by a special operating system (or certain components thereof) running on the computer device of FIG. 55 (a), according to an embodiment of the present invention.
FIG. 57a shows in more detail how the last step of the method of FIG. 56 might look like.
FIG, 57b is a variation of FIG. 57a.
FIG. 58 shows a variation of the method of FIG. 56, as can be performed by an improved control system, according to an embodiment of the present invention.
FIG. 59 shows a variation of the computer system of FIG. 55, according to an embodiment of the present invention. FIG. 59 (a) and FIG. 59 (b) show a computer system with a special operating system according to an embodiment of the present invention with two mouse cursors. In FIG. 59 (b) the first pointing device is moved. In FIG. 59 {b} the second pointing device is moved. FIG. 59 (c) is a schematic representation of a so-called "Z-order" of windows or image planes and a "cursor plane" as may be used in the computer system of FIG. 59 (a). In FIG. 59 (a) and FIG. 59 (b) uses three bitmaps, including only one native mouse cursor. FIG. 59 (d} and FIG. 59 (e) show two bitmaps that can also be used.
It is noted that the timing diagrams shown herein are high-level timing diagrams, and many details have been omitted, which are not relevant to an understanding of the principles of the present invention.
Detailed Description of Illustrative Embodiments The present invention will be described with reference to particular embodiments and with reference to particular drawings, however, the invention is not limited thereto, but is limited only by the claims.
The figures are only schematic and not limiting.
In the figures, the dimensions of some parts may be exaggerated and not to scale for illustrative purposes.
The dimensions and the relative dimensions sometimes do not correspond to the current practice of the invention.
Furthermore, the terms first, second and the like are used in the specification and in the claims to distinguish similar elements and not necessarily to describe an order, neither in time, nor spatial, nor in order, or in any other way.
It is to be understood that the terms used in this manner are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are adapted to operate in a different order than described or illustrated herein.
It should be noted that the term "comprises", as used in the claims, is not to be construed as being limited to the means described below; this term does not exclude other elements or steps.
It can thus be interpreted as specifying the presence of the stated characteristics, values, steps or components to which reference is made, but does not exclude the presence or addition of one or more other characteristics, values, steps or components, or groups thereof.
Thus, the scope of the expression "a device comprising means A and B" should not be limited to devices consisting only of components A and B.
It means that with respect to the present invention, A and B are the only relevant components of the device.
Reference throughout this specification to "one embodiment" or "an embodiment" means that a specific feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention.
Thus, occurrences of the terms "in one embodiment" or "in an embodiment" in various places throughout this specification may not necessarily all refer to the same embodiment, but may do so.
Furthermore, the specific features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.
Similarly, it should be appreciated that in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof for the purpose of streamlining disclosure and aiding the understanding of one. or more of the various inventive aspects.
In any event, this method of disclosure should not be construed as reflecting an intention that the invention requires more features than explicitly stated in any claim. Rather, as the following claims reflect, inventive aspects reside in less than all of the features of a single prior disclosed embodiment. Thus, the claims following the detailed description are hereby explicitly incorporated into this detailed description, with each self-contained claim as a separate embodiment of this invention.
Furthermore, while some embodiments described herein include some, but not other features included in other embodiments, combinations of features of different embodiments are intended to be within the scope of the invention, and constitute various embodiments, as would be understood by those skilled in the art. . For example, in the following claims, any of the described embodiments can be used in any combination.
Numerous specific details are highlighted in the description provided herein. Either way, it is understandable that embodiments of the invention can be practiced without these specific details. In other instances, well known methods, structures and techniques have not been shown in detail to keep this description clear.
In this document, the term "alpha blending" or "blending" (of images or bitmaps) is used synonymously. In the context of graphic overlay, there is an upper image (which has a higher Z order) and a lower image or lower image (which has a lower Z order). The blending can be performed in hardware and / or software, and is based on a parameter called “alpha blending value” à, which refers to the level of blending of the two images, where a = 0% means that the original image (with the lower Z-order) is not mixed with the overlay image (or that the overlay image is completely transparent), and where a = 100% means that the resulting pixel value is that of the overlay image (or that the overlay image is completely opaque (except for fully transparent pixels) The alpha blend value can be expressed on a scale from 0 to 255, where a = 255 corresponds to a = 100% and a = 0 corresponds to a = 0%. For example, alpha gene can be used to fade in or fade out of a graphic image In the context of the present invention, there may also be a second parameter involved in the blending, called "TransparentColorValue." If a pixel of the overlay image has this predetermined value, this pixel is treated d as “completely transparent”. Fully transparent pixels are typically used on the outline of a bitmap, for example to display a rectangular image with rounded edges. "Alpha gene" and "fully transparent pixels" are both well known in the art.
In this document, the expressions "word processor" or "text editor" should not be interpreted too narrowly, because many so-called word processors (such as Microsoft Word) also have drawing capabilities. Likewise, the term "spreadsheet" or "worksheet" should not be interpreted too narrowly, as many so-called spreadsheet applications, eg Microsoft Excel, also have graphical capabilities. The phrase "the overlay window is configured in click-through mode" or "the overlay window is configured in" pass-through mode "or in" pass-through mode "means that the overlay window is configured in such a way that events (events) from input devices (such as e.g. a mouse, a keyboard, a trackball, a touchpad, a stylus, a joystick, etc.) sent by the operating system to one or more underlying application windows or objects thereof, despite the higher Z-order of the overlay application, and despite the native mouse cursor being located above the area defined by the overlay window. For example, the message "passed to lower windows" applies to events triggered by a button press, and a button release, and a button click, and a mouse wheel scrolling event, and pointing device movements In the context of the present invention, the o verlay application typically has one window configured in click-through mode, and optionally but preferably also includes a second, substantially opaque window with user interface elements, not configured in click-through mode.
In this document, the expression "the window is configured in non-click-through mode" means that the window is "not configured in click-through mode".
In this document, the term "work area" refers to a portion of the pixel area of a screen, namely the area that will typically be occupied by an application in the event that it is maximized. In a Microsoft Windows environment, the "work area" means the entire pixel area or area of the desktop minus the so-called "task bar", which is typically located at the bottom of the screen.
In this document, the terms "mouse pointer" and "mouse cursor" are used synonymously, and refer to a symbol or graphic image on a computer monitor or other display device that represents movements of a pointing device, e.g., a mouse, touch pad, or stylus pen. This pointer or cursor can take various forms, such as for example an "arrow" or a "crosshair" (which looks like a roman number "I") or as a "hand" etc.
In this document, the terms "display screen" and "monitor" and "screen" are used synonymously.
In this document, the terms line "or" bar "or" line segment "are used synonymously and refer to an elongated, for example a rectangular, object, optionally with rounded edges.
In this document, the expression "the mouse cursor or pointer follows movements of the pointing device" means that the position of the cursor or pointer is adjusted in accordance with movements of the pointing device.
In this document, the terms "first maintained position" and "position related to movements of the first pointing device" mean the same. This position is also referred to herein as Pos1 or the coordinates (x1, y1) or the coordinates (xR, yR).
In this document, the "first pointing device" usually refers to the pointing device operable by the right hand of a user, unless explicitly stated otherwise.
In this document, the terms "second maintained position" and "position related to movements of the second pointing device" mean the same. This position is also referred to herein as Pos2 or the coordinates (x2, y2) or the coordinates (xL, yL).
In this document, the "second pointing device" usually refers to the pointing device operable by the left hand of a user, unless explicitly stated otherwise.
In this document, the phrase "shift focus between two documents" means to shift focus between two locations on the screen, e.g., between a first window associated with a first application (e.g., a PDF viewer) that contains a portion of a first document. and a second window associated with a second application (e.g., a word processor) that displays a portion of a second document, unless explicitly stated otherwise.
In this document, the term "host" means a "computer device" unless explicitly stated otherwise.
In this document, the terms "signal" and "message" sent by a pointing device mean the same.
In this document, the term “underlining” or “highlighting” a text fragment means the apparent or virtual underlining or highlighting of the text fragment by means of an overlaid object.
In this document, the term "motion sequence" refers to a sequence or series of at least two or at least three motion or displacement messages (as shown, e.g., in FIG. 47), or to a data structure corresponding thereto. These are also referred to herein as "false" movements because they do not correspond to actual movements of the pointing devices.
In this document, the term "button message" refers to a "button-pressed message and / or a button-released message" without specifying which button it is (e.g. a left mouse button, or a right button of a mouse) and without specifying whether it is a button press or a button release.
When hereinafter referred to as "scroll message", it may be a "scroll-up message" or a "scroll-down message", or both, but it is not always necessary to make a distinction in the description. . Likewise, the term "scroll command" is sometimes simply used instead of the "scroll up command" or the "scroll down command".
By a "zero displacement" is meant a displacement with dx = 0 and dy = 0. By a "zero displacement message" is meant a displacement message with dx = 0 and dy = 0.
In certain embodiments of the present invention, a "first / second position" is tracked related to movements of a first / second pointing device (e.g., a first / second mouse). This tracked first / second position is also referred to as "first / second pointer position". It corresponds to a first / second screen position (English: display position).
If there is only one mouse cursor (e.g., the native mouse cursor provided by the O / S), it can occupy either the first or second position (or hover position). This position is referred to as the "cursor position." If there are two mouse cursors, then there is a first and second cursor position.
As described in the background section and as shown in FIG. 1 to FIG. 4, there are many occasions when a computer user would like the computer system to have a mechanism that allows the user to read and / or write and / or scroll and / or click in two different locations on a screen, while today, in all major personal computer operating systems with a graphical user interface, only a single visible object called "mouse pointer" or "mouse cursor" is supported.
While the present invention is not limited to "Windows-compatible" computers running a version of the "Windows" operating system from Microsoft Corporation (Redmond, USA), the present invention will be explained primarily with such computers in mind, in order to describe the description. to simplify. But the principles of the present invention will also work for other computers with an operating system O / S and a graphical user interface GUI that provides only a single mouse pointer, such as, for example, mac OS from Apple Inc., or Linux.
While the present invention is not limited to working with two "mouse devices" as pointing devices, the present invention will be explained with two mouse devices in mind, in order to simplify the description. However, the principles of the present invention will work with other pointing devices as well, such as, for example, a touch pad or a trackball or a stylus or the like.
The problems faced by a translator or proofreader or writer or transcriber in the cases shown in FIG. 1 to FIG. 4, are related to a feature (or shortcoming) of operating systems with a GUI that only maintain a single mouse cursor or mouse pointer. From a commercial point of view, one might think that there is a simple solution to these problems, which is to connect a second pointing device to the computer device and hope that somehow a second mouse cursor will appear showing movements of the second pointing device. follows. Unfortunately, classic personal computer operating systems such as Windows and macOS do not work that way because they only track a single cursor position, and although multiple pointing devices can be physically connected to the computer device, all signals coming from these pointing devices are moved to one cursor position. led. For example, when two USB mouse devices are connected to a laptop computer running Windows 7 or Windows 10, the Windows native mouse cursor can be moved, for example, to the left by moving each of these mouse devices to the left. No second mouse cursor will automatically appear.
In contrast, the present invention provides a computer system that supports two pointing devices and provides two or at least two visible objects, which are individually movable by the two pointing devices, e.g., a first mouse device and a second mouse device, or only one visible object (namely the native mouse cursor) that selectively occupies one of two maintained positions.
While it would be technically possible to develop a new operating system with inherent support for multiple pointing devices and multiple cursors, such a solution is not a realistic option, at least not for third parties. The inventors of the present invention have therefore sought to find a solution based on an existing operating system O / S with a graphical user interface GUI that provides a single native mouse cursor, and extend or improve its functionality, ideally in such a way that most or all existing applications can benefit from this without modifying these applications.
As will be further explained in more detail, the present invention proposes to partially solve the problem in a "graphics overlay application" (see, e.g., FIG. 5f, FIG. 11, FIG. 12, FIG. 23a, FIG. 24, FIG. 25, FIG. 45, FIG. 46, FIG. 53, FIG. 54) and partially in the driver layer (see e.g. FIG. 5f, FIG. 11, FIG. 12, FIG. 45, FIG. 46) and / or in a special pointing device (see, e.g., FIG. 23a, FIG. 24, FIG. 53) and / or in a dedicated hub (see, e.g., FIG. 25, FIG. 54). An underlying principle of the present invention is that the overlay application maintains two positions, one for each pointing device, and that the overlay application makes the native mouse cursor of the O / S occupy one of these two positions. This offers the great advantage that the look and appearance of this mouse cursor usually remains under control of the O / S and the GUI and the other applications (except when the mouse cursor moves over a user interface window of the overlay application). Therefore, the solution provided by the present invention allows the mouse pointer to change shape depending on the context of the application under the overlay application, e.g. to "an arrow pointer" or a "pointer with a | dash" or a " hourglass "etc. This advantage should not be underestimated, as the shape of the mouse cursor often provides" visual cues "to the user that would otherwise be lost.
Referring now to the figures.
FIG. 1 to FIG. 4 are already described in the background section, and illustrate several practical examples when a single user (particularly a professional user) wants to use a personal computer with two pointing devices, one operable with his / her dominant hand, the other operable with his / her not. dominant hand.
In order to quantify the requirements of a potential two pointing device solution, it is assumed that the user of FIG. 1 to FIG. 4 is right-handed, and that the computing device of FIG. 1 to FIG. 4 is connected to two pointing devices: a "right mouse device" for the right hand, and a "left mouse device" for the left hand, then: (1) the user of FIG. 1 and FIG. 2 and FIG. 3 move and / or click and / or scroll the right mouse device, and wants to move the left mouse device, but does not necessarily need to click or scroll the left mouse device; (2) Alternatively, the user of FIG. 1 move and / or click and / or scroll the right mouse device, and wants to move and / or drag the left mouse device (i.e. press and move and release) and / or scroll; (3) the user of FIG. 4 move and / or click and / or scroll the right mouse device, and wants to move and / or click the left mouse device, but does not necessarily need to scroll the left mouse device.
FIG. 5 shows a computer system 500 according to an embodiment of the present invention, comprising a computer device 501 and two pointing devices 503R, 503L and a display screen 504 and optionally a keyboard 502. The computer device 501 is connected to a first pointing device 503R and to a second pointing device 503L, and executes an O / S and GUI (e.g. Windows 10 from Microsoft) that provides a native mouse pointer / mouse cursor 505 (shown here in the form of a white arrow), and executes an overlay application according to an embodiment of the present invention, and optionally executes one or more other applications, such as e.g. a word processor, a PDF document viewer, a web browser, etc. The overlay application maintains two positions Posil, Pos2 in accordance with movements of the first and second pointing device respectively, and repeatedly overwrites the position of the native mouse pointer / mouse cursor 505 to match and with movements of the first pointing device 503R. The overlay application allows the native mouse pointer / mouse cursor 505 of the O / S and GUI to be displayed as a first visible object instead of hiding it. The overlay application also provides a second visible object 506L that moves in accordance with movements of the second pointing device 503L.
FIG. 5 (a) shows movements of the first visible object 505, being the native mouse cursor provided by the O / S and the GUI, in accordance with movements of the first pointing device
503R. The second visible object 506L shows the second maintained position Pos2, but is stationary in FIG. 5 (a), because in this example the second mouse pointer 503L is not moved.
FIG. 5 {b} shows movements of the second visible object 506L provided by the overlay application, in accordance with movements of the second pointing device 503L. The first visible object 505 shows the first maintained position Pos1, but is stationary in FIG. 5 (b), because in this example the first mouse pointer 503R is not moved.
FIG. 5 (c) is a schematic representation of a so-called "Z-order" of five windows or image planes 581-585 and a "cursor plane" 586 as may be used in the computer system 500 of FIG. 5 (a) and FIG. 5 ( b).
The plane 581 may be a background plane (known in the case of Windows as the "desktop"), the image plane 582 may be the visible representation of a window associated with a text editor application, for example, the image plane 583 may be the visible representation of a For example, the window associated with a PDF viewer application, the image plane 584 may be the visible representation of a window associated with the graphics overlay application of the present invention, the cursor plane 586 is the image plane associated with the native mouse cursor provided by the O / S and the GUI, the shape of which can be changed by an underlying application. The graphical overlay application of the present invention may also have an optional user interface window 585, an example of which will be seen in FIG. 28 to FIG. 31 are described. The user interface window 585 is not configured in click-through mode, but is configured in non-click-through mode, that is, its visible components (e.g., buttons, radio buttons, checkboxes, etc.) are able to receive (among other things) mouse clicks.
The window 584 of the overlay application can have a size equal to the visible screen, or equal to the area of the desktop plane 581, or equal to the area of the so-called work area (i.e. the area of the screen minus the area occupied by the so-called toolbar "), or some other suitable size. The window 584 includes at least one visible object 506L, for example in the form of a rectangular object. The object 506L may be opaque or semi-transparent. The object 506L may have monochrome pixels, for example, red or blue pixels, or may have a bitmap with a color gradient, etc. The other pixels 594 of the window 584 may be fully transparent pixels. Alternatively, the window 584 may be configured as a semi-transparent window, for example, with an alpha transparency in the range from 1% to 99%, for example, in the range from 5% to 95% In this case the pixels of the window 584 may have a color, for example is a monochrome color, but the window 584 may also include a so-called "perforated bitmap" or a "texture bitmap" or a "perforated texture bitmap". These aspects are described in more detail in patent application BE2017 / 5891 filed December 4, 2017 by the same applicant, entitled "METHOD, DEVICE AND COMPUTER PROGRAM FOR OVERLAYING.
A GRAPHICAL IMAGE ", and in patent application PCT / EP2018 / 075836 of the same title, filed December 24, 2018, which documents are incorporated herein by reference in their entirety, and further referred to herein as the co-pending" perforated bitmap application. (n) ". In the event of a conflict between the present document and the co-pending application (s), the present document prevails. The present invention can be seen as a variant of the patent application BE2017 / 5895 filed December 4, 2017 , and from patent application PCT / EP2018 / 075838 filed September 24, 2018, entitled "METHOD AND DEVICE AND SYSTEM FOR PROVIDING DUAL MOUSE SUPPORT", which documents are also incorporated herein by reference in their entirety.
In the example of FIG. 5 (c) the graphics overlay application has two windows: - a window 584 that is configured in click-through mode and is mainly transparent or semi-transparent (e.g., with an alpha transparency in the range of 5% to 95%), - and a another window 585 (also called user interface window) which is preferably not configured in click-through mode and which is mainly opaque.
FIG. 5 {d} shows a high-level flow diagram of a computer-implemented method 570 according to an embodiment of the present invention, for providing a first and a second visible object 505, 506L movable separately by a first pointing device 503R and a second, respectively. indicating device 503L.
The method may be implemented in an overlay application 551 (see FIG. 5f) that may be run on the computing device 501 (see FIG. 5a), which computing device 501 also includes an operating system O / S with a graphical user interface GUI that contains a single native mouse cursor 505 provided. The computer device 501 is connected to the first and second pointing devices 503R, 503L. The method maintains two positions Pos1 and Pos2 (see also FIG. 5f). The method shown in FIG. 5d includes the following steps: a) configuring 571 the operating system O / S with graphical user interface GUI to provide the native mouse cursor 505 as the first visible object, e.g. by allowing the O / S to display the mouse cursor or by not hiding the mouse cursor explicitly; b) providing 572 a graphics overlay window 584 including the second visible object 506L; c) configuring 573 the graphics overlay window 584 in click-through mode; e) adjusting a position of the native mouse cursor 505 in accordance with movements of the first pointing device 503R, and adjusting a position of the second visible object 506L in accordance with movements of the second pointing device 503L.
In one embodiment, the position of the first visible object 505 is updated at the same frequency as the position of the second visible object 506L.
In another embodiment, the position of the first visible object 505 is updated at a different frequency than the position of the second visible object 506L. For example, the position of the native mouse cursor can be updated in a message processing routine (see, for example, FIG. 5g), and the position of the second visible object can be updated, for example, based on a timer interrupt.
FIG. 5 {e) shows a more detailed example of a computer-implemented method according to an embodiment of the present invention, which can be seen as a special case of the method shown in FIG. 5 (d). The method of FIG. 5 (e) further includes step d): d) configuring 574 the operating system O / S to send raw input messages; In case the O / S is a Microsoft Windows variant, this step may include registering the application at the O / S to receive raw input data, e.g. in the form of input messages known as WM_INPUT messages, e.g. using the WinAPI function RegisterRawlnputDevices (). It is noted that applications do not automatically receive raw input messages.
And wherein step e) comprises: repeatedly performing: f) receiving 575 a raw input message; g) determining whether the input raw message is related to a movement of the first or second pointing device 503R, 503L, and if the result of this test is true, obtaining displacement information dx, dy related to the displacement, and updating the corresponding first or second position Pos1, Pos2; and if the result of this test is false, skipping step h); h) adjusting 577 the position of the native mouse cursor 505 based on the first maintained position Pos1, or the second maintained position Pos2, or the position Pos1 or Pos2 associated with the pointing device that was most recently operated. In the example of FIG. 5 (a) and FIG. (B) the position of the native mouse cursor 505 is always updated in accordance with movements of the first pointing device 503R, but as will be further explained, this position can also be updated in another way.
And wherein step e) further comprises: i) repeatedly adjusting 578 a position of the second visible object 506L in accordance with the second position (Pos2). This step i) can also be performed asynchronously with respect to steps f) to h), for performance reasons. Alternatively, step i) can be performed after each execution of step h).
Step g) in combination with step h) is also referred to herein as the "position update routine", it is part of the message processing routine. Step i) is also referred to herein as "article repositioning routine".
FIG. 5 {f) shows a simplified high-level block diagram of hardware and software components located in the computer system 500, comprising a computer device 501 with a special overlay application according to an embodiment of the present invention, and two classic pointing devices M1, M2.
The overlay application maintains two pointer positions: a first position Pos1 which has coordinates (x1, y1) and a second position Pos2 which has coordinates (x2, y2). A message processor 552 of the overlay application 551 will update the first and second position Pos1, Pos2 based on movements of the two pointing devices, e.g., as discussed further in FIG. 5 (g).
The graphics overlay application of the system 500 repeatedly adjusts the position of the native mouse cursor 505 to the first maintained position Pos1, according to movements of the right mouse device 503R, and repeatedly adjusts the position of the second visible object 506L to match with the second maintained position Pos2, in accordance with movements of the left mouse device 503L, or vice versa.
FIG. 5 (g) shows some lines of pseudo code to illustrate a simplified version of a message processing routine as can be used in the overlay application 551 provided in the computing device 501 of FIG. 5 (a) and FIG. 5 (b) is performed. The message processor 552 is triggered by incoming raw messages from input devices (such as, e.g., keyboard and / or pointing devices) in a manner well known in the art. The message processor may obtain or request further information related to each message, for example to obtain a device handle to determine which pointing device sent the message, and / or to obtain a horizontal and vertical displacement value dx, dy as an indication of a movement in the horizontal and / or vertical direction, and / or to get a value indicating how much a scroll wheel has scrolled, etc.
As can be understood from the pseudo code, if the message was sent by the left pointing device 503L, the second position Pos2 will be updated using the displacement values dx, dy, and the position of the native mouse cursor 505 will be set (or reset) are related to the first position Pos1 to the right pointing device 503R. If the message was sent by the right pointing device 503R, the first position Pos1 will be updated using the displacement values dx, dy, and the position of the native mouse cursor 505 will be adjusted to move accordingly. Tests have shown that this routine also correctly handles simultaneous movements of the left and right pointing device 503L, 503R.
In the example shown in FIG. 5 (g) the X and Y coordinate of the first and second position Pos1, Pos2 are incremented by the displacement values dx, dy as received from the pointing devices, but in practice, the displacement values dx, dy may be transformed first, linearly or not -linear, to make the movements less or more sensitive (see the radio buttons "Speed Left / Right Mouse" [English: Speed Left Mouse, Speed Right Mouse] and the "Speed Curve" [English: Speed curve] in FIG. 31a) . As shown in FIG. 31 (b) and FIG. 31 (c) to be discussed further, other adjustments of the displacement values are possible.
FIG. 5 {h} shows some lines of pseudo code to illustrate a simplified version of a routine that updates the position of the second visible object 506L, in the example being a rectangle, but the present invention is not limited thereto. This routine can be based on a timer interrupt. In this way, updating the position of the second visible object 506L outside of the message processor can be handled, which may be preferred for performance reasons. The period of the timer can be selected, for example, in the range of 1 ms to 250 ms, preferably in the range of about 1 ms to about 100 ms, for example equal to about 1 ms or about 5 ms or about 10 ms or about 20 ms or about 25 ms or about 30 ms or about 35 ms or about 40 ms or about 50 ms or about 75 ms or about 100 ms. The value of this period is not critical. The smaller this time period value, the faster the response, but the higher the CPU load. The greater this time period value, the slower the response, but the lower the CPU load. The skilled person can easily find a suitable value by routine experimentation. Some embodiments may allow the end user to select a preferred timer value.
FIG. 5 (a) to FIG. 5 (h) illustrate a first embodiment of the present invention, particularly suitable for right-handed people. The O / S and GUI provide the native mouse cursor 505, but the overlay application repeatedly adjusts its position such that the position is defined only by movements of the right mouse device 503R. The right mouse device 503R can be moved, its buttons can be pressed or released or clicked, and its scroll wheel can be scrolled as in the prior art. In addition, the overlay application shows a second visible object 506L, which in the example of FIG. 5 (a) is in the form of a line or rectangle, and the overlay application repeatedly adjusts its position such that the position is defined only by movements of the left mouse device 503L. This embodiment successfully addresses the data shown in FIG. 1 to FIG. 3 problems shown. The user can use his non-dominant hand to move the second object 506L to underline or highlight text he or she wants to read, and use his dominant hand to edit a document, select text, choose menus, etc. , using the native mouse cursor 505. Thus, the overlay application improves the O / S and GUI by providing one additional visible object 506L that is movable by the left pointing device 503L, while retaining most or all of the functionality of the native mouse cursor. For example, the additional visible object 506L can be used to underline or highlight text fragments.
In one embodiment, the overlay application may be a dedicated application that displays only the specific behavior in FIG. 5 (a) and FIG. 5 (b), wherein the native mouse cursor 505 is moved by the right pointing device, and a second visible object 506L in the form of a line movable by the left pointing device, in which case the user interface window 585 may be omitted.
In another embodiment, the overlay application can implement several options, and a user can select a mode from a variety of modes. In this case, a user interface window is useful, see for example the exemplary user interface of FIG. 28 (a), e.g., mode 2860 (d), and the exemplary user interface of FIG. 30, e.g. mode 3060 (d). The "big right mouse" shown on the left of FIG. 28 (a) is a hint to the user that the right mouse is the dominant mouse for the 3060 modes, that is, the "right mouse device" controls the native mouse cursor 505. The "little left mouse" is a hint to the user that the left mouse device is the non-dominant mouse driving the second visible object, in the case of mode 2860 (d) being a line or rectangle.
FIG. 6 (a) to FIG. 6 (c) shows a computer system 600 according to another embodiment of the present invention, which can be viewed as a variant of the computer system 500 of FIG. 5, wherein a computing device 601 is connected to a first and second pointing device 603R, 603L, and running an O / S and GUI (e.g. Windows 10) that provides a native mouse pointer 605 (e.g. in the shape of a white arrow), and executes an overlay application according to an embodiment of the present invention. The overlay application maintains a first and a second position Pos1, Pos2 in accordance with movements of the first and second pointing device 603R, 603L, respectively, and adjusts the position of the native mouse pointer 605, e.g. overwrites it, in accordance with movements of the first pointing device 603R. The overlay application provides a second visible object 606L that moves in accordance with movements of the second pointing device 603L, and also provides a third visible object 606R that moves in accordance with movements of the first pointing device 603R. In the example of FIG. 6, the second and third objects 606L and 606R are rectangles of the same size and color, but this is not an absolute requirement, and the two visible objects may be of different size and / or different shape and / or different color. Thus, the overlay application used in the computer system 600 of FIG. 6 improves the O / S and GUI by providing two additional viewable objects 606L, 606R that are movable by the two pointing devices 603L, 603R, and can be used, for example, to underline or highlight text fragments of classic applications running on the computer. executed, for example: a word processor, a web browser, a slideshow, a PowerPoint presentation, a PDF document viewer, etc. The objects 606L, 606R can be opaque or semi-transparent or can contain a “perforated bitmap” as described in the co-pending being "perforated bitmap application (s)".
FIG. 6 (a) shows movements of the first visible object 605 (the native mouse cursor) and movements of the third visible object 606R in accordance with movements of the first pointing device 603R (e.g., the right mouse).
FIG. 6 (b) shows movements of the second visible object 606L in accordance with movements of the second pointing device 603L (e.g., the left mouse).
FIG. 6 {c} shows an exemplary object repositioning routine that can be used in the overlay application to move the two visible objects 606L, 606R, in pseudo code. Simply stated, the coordinates of the object 606L (called rectangleL) are updated according to the second position Pos2 with coordinates xL, yL associated with the second mouse device 603L, and the coordinates of the object 606R (called rectangleR) are updated according to the first position Pos1 with coordinates xR, yR associated with the first mouse device 603R. These steps are performed repeatedly, for example on the basis of a timer.
It is noted that, in FIG. 6 (a) and FIG. 6 (b), the first visible object 606R is displayed on the right side of the screen 604, and the second visible object 606L is displayed on the left side of the screen 604, but it is not absolutely required, and both objects can be anywhere on the screen. screen, as shown, for example, in FIG. 37. In practice, however, a user will typically find it more intuitive to position the object related to the left pointing device on the left side of the screen. In other embodiments, the movement of the second and optionally further object (s) 3906R, 3906L is limited, as shown in FIG. 39, which will be discussed later, but even then, the movement of the native mouse cursor 3905 is preferably not limited.
The computer system 600 of FIG. 6 can be used, for example, by proofreaders to apparently underline text portions in two documents displayed side by side on the screen 604, without actually altering the content of the documents themselves. It is a significant advantage that text fragments can appear to be underlined in two different documents at the same time. This can drastically reduce the user's cognitive burden when switching focus back and forth between the two documents. It is an important advantage that the text fragments can be independently underlined (unlike, for example, word processors that allow single-synchronous navigation through two text documents), not only because sentences in one language can be longer or shorter than sentences in another language, but also because sentences in one language can be longer or shorter than sentences in another language. the two documents can have different formatting, eg different layout and / or different font and / or different line spacing, etc. It is an advantage that "the second mouse and the second object" can be used in combination with almost any underlying application, e.g. a word processor, a PDF document reader, a spreadsheet, a web browser, a PowerPoint application, etc. without having to change any of these applications. For example, the object 606L can be moved across a PDF document, and the object 606R can be moved across a text document, and vice versa. In the example of FIG. 6 (a) and FIG. 6 (b), the native mouse cursor 605 moves with the object 606R.
The user can use the native mouse cursor 605 to click and / or drag anywhere on the screen and / or choose menu options, for example to select text in the text document which can then be erased or replaced with other text using the keyboard 602, or to open new applications in the taskbar (not explicitly shown) in the same way as can be done without the graphics overlay application.
In the exemplary user interface of FIG. 28 (a), the behavior of FIG. 6 (a) and FIG. 6 (b) correspond to mode 2860 (h). (the word "can" is used because several variations are possible, for example for the message processor, as will become further apparent).
FIG. 7 shows a computer system 700 according to another embodiment of the present invention, which can be viewed as a variant of the computer system 600 of FIG. 6, wherein a computing device 701 is connected to a first pointing device 703R and a second pointing device 703L, and executing an O / S and GUI that provides a native mouse pointer or mouse cursor 705 (e.g., a white arrow), and an overlay application according to an embodiments of the present invention, which provides second and third visible objects 706L, 706R.
The overlay application maintains a first and a second pointer position Pos1, Pos2 in accordance with movements of the first and second pointing device 703R, 703L, respectively, and adjusts the position of the native mouse pointer / cursor 705, e.g. overwrites it, accordingly with movements of the pointing device that was most recently moved.
Thus, if the right pointing device 703R is moved and the left pointing device 703L is stationary, as shown in FIG. 7 (a), the native mouse cursor 705 tracks movements of the right pointing device 703R, and the buttons and / or the scroll wheel of the right pointing device 703R may be used. If at some other point in time the left pointing device 703L is moved and the right pointing device 703R is stationary, as shown in FIG. 7 (b), the native mouse cursor 705 tracks movements of the left pointing device 703L, and the buttons and / or the scroll wheel of the left pointing device 703L can be used. As can be understood from FIG. 7 (a) and FIG. 7 (b) the native mouse cursor tracks movements of the pointing device being moved, until the other pointing device is moved, causing the native mouse cursor to jump to the other maintained position. In this way, the user can use the native mouse pointer at two different locations of the screen, albeit at different points in time. Or expressed in other words, in this embodiment, the native mouse cursor 705 provided by the O / S and GUI is temporarily associated with one of the pointing devices 703L, 703R at any point in time, so the native mouse pointer 705 becomes " divided in time "between the two indicating devices.
FIG. 7 (c) is a high-level flow diagram showing a computer-implemented method according to an embodiment of the present invention, as can be used in the computer system of FIG. 7 (a).
FIG. 7 (c) depicts a computer-implemented method for providing a first visible object 705 and a second visible object 706L and a third visible object 706R. The third visible object 706R is movable by a first pointing device 703R. The second visible object 706L is separately movable by a second pointing device 703L.
The first visible object 705 is selectively movable by one of the first pointing device 703R and the second pointing device 703L, depending on which one was most recently moved and / or operated. In FIG. 7 (a) the first pointing device 703R is moved, and the native mouse cursor 705 and the third visible object 703R both move in accordance with movements of the right pointing device, or more specifically, in accordance with a first position Posi. In FIG. 7 (b) the second pointing device 703L is moved, and the native mouse cursor 705 and the second visible object 706L both move in accordance with movements of the left pointing device 703L. It can be understood that the native mouse cursor 705 moves gradually as long as only one pointing device is moved, but typically jumps suddenly from Pos1 to Pos2 or vice versa when the other pointing device is moved.
FIG. 7 (c) shows a flow diagram illustrating a computer-implemented method comprising the following steps: a) configuring, e.g. allowing an operating system O / S with a graphical user interface GUI to provide a native mouse cursor 705 as a first visible object ; b) providing a graphics overlay window that includes a second visible object 706L and a third visible object 706R; c) configuring the graphic overlay window in click-through mode;
e) adjusting 774 a position of the native mouse cursor 705 in accordance with movements of one of the first pointing device 703R and the second pointing device 703L, depending on which movement was most recent, and adjusting 775 a position of the second visible object 706L in accordance with movements of the second pointing device 703L, and adjusting a position of the third visible object 706R in accordance with movements of the first pointing device 703R.
In a variation of the embodiment shown in FIG. 7 (a) to FIG. 7 (c), no two horizontal stripes are shown, but only the native mouse cursor 705 is shown, and repositioned as described above. Thus, in this case, steps 772 and 775 can be omitted (hence the dashed line representation).
In a further variant of this embodiment, step c) is also omitted. Such a method can be performed, for example, by a custom application (e.g. one of the applications 562 of FIG. 5f) which itself has provisions for supporting two mice, without having to use the overlay application 551. This variant can be used. can also be executed by a module that can be added to the operating system, to provide an improved operating system that is actually based on one native mouse cursor that can jump. In that case, step 771 can also be omitted, because it is already provided in the operating system.
FIG. 7 {d} is a more detailed flow chart showing a computer-implemented method 770 according to an embodiment of the present invention, as may be used in the computer system of FIG. 7 (a). The method 770 includes the following steps: a) configuring 771, e.g., allowing an operating system O / S with a graphical user interface GUI to provide a native mouse cursor 705 as a first visible object; b) providing 772 a graphics overlay window that includes a second visible object 706L and a third visible object 706R; c) configuring 773 the graphical overlay window in click-through mode; d) configuring 774 the O / S to send raw input messages, and providing a message processor to process these raw input messages; e) repeatedly performing the following steps: f) receiving 775 a raw input message; g) if the received input raw message is related to a movement of the first / second pointing device 703R, 703L, obtaining 776 movement information dx, dy and updating a first and second position Pos1, Pos2, respectively;
h) adjusting 777 a position of the native mouse cursor 705 in accordance with the first or second position Pos1, Pos2 depending on the most recent input message; and repeatedly performing the following step: i) adjusting 778 a position of the second and third visible objects 706L, 706R in accordance with the first position Pos1 and the second position Pos2, respectively.
In a variation of this embodiment, which can be performed e.g. in an overlay application where no two horizontal stripes are displayed, but where only the native mouse cursor 705 is displayed and repositioned, part of step 772 can be omitted , as well as step 778.
In a further variant of this embodiment, which can e.g. be implemented in a module added to the control system, steps 772 and 773 and 778 can be omitted altogether. Steps 771 and 774 and 775 are in principle already present in the operating system, but could possibly be implemented in a different way (e.g. a more efficient way) in the operating system itself. This module can show e.g. a second cursor plane (e.g. with a second white arrow or with a second black or blue or red arrow), with a Z-order higher than that of the applications, e.g. like the cursor plane 586 of FIG. 5c.
FIG. 7 (e) shows an exemplary position updating routine (as part of the message processing routine), in pseudo code, that can be used to update the position of the two horizontal lines. As can be seen, the coordinates (xR, yR) and (xL, yL) of the first and second position Pos1, Pos2 are updated in accordance with movements of the first and second indicating device 703R, 703L, respectively (albeit indirectly, using a device handle that assigns the O / S to each pointing device), and the position of the native mouse cursor 705 is adjusted depending on the most recent motion message.
An advantage of this embodiment is that a user can use two mouse devices to simultaneously display two visible objects, e.g. two lines or two rectangles for underlining or highlighting text fragments at two different locations of the screen, and in addition, the user can also Click or scroll the mouse device most recently moved. This can be used, for example, for scrolling up / down the respective documents when proofreading.
FIG. 8 depicts another computer system 800 according to an embodiment of the present invention, which can be viewed as a variation of the computer system 500 of FIG. 5, the second visible object 806 having the shape of a black arrow, but of course the invention is not limited thereto, and an arrow of another color may also be used, but the drawing is not limited to an arrow, and other shapes as well. can be used, see for example FIG. 34 where two bitmaps with an image of a hand are used. As shown in FIG. 8 (c) includes the overlay application used in the computer system 800 of FIG. 8 is a click-through window 884 similar to the window 584 of FIG. 5 (c), but wherein the rectangular object 506L has been replaced by a visible object (e.g., an "image component") containing a suitable bitmap. The image component 806L may contain a bitmap that contains pixels representing a black arrow surrounded by fully transparent pixels 895 (only some of which are shown), which is suitable in the event that the other pixels 894 of the window 884 are also fully transparent pixels. In case the window 884 is configured as a semi-transparent window and the pixels 894 are of a particular color (e.g., light gray), then the black arrow is preferably surrounded by semi-transparent pixels 895 of that same color (e.g., light gray). light gray). In the event that the window 884 is configured as a semi-transparent perforated bitmap or a semi-transparent perforated texture bitmap as described in more detail in the co-pending "perforated bitmap application (s)", the black arrow is preferably also surrounded by the same or similar perforated bitmap In the latter case, the object repositioning routine will preferably locate the movable object 806L such that the perforated background of the movable image is aligned with the perforated bitmap of the window 884, which in the case of a 2x2 bitmap with a chessboard pattern can be implemented by positioning the movable object 806L, for example, only at locations with even X and even Y coordinates, or only at locations with odd X and odd Y coordinates, depending on where or how the pattern of pixels 894 is located.
FIG. 8 (a) shows movements of the first visible object 805 in accordance with movements of the first pointing device 803R.
FIG. 8 (b) shows movements of the second visible object 806 in accordance with movements of the second pointing device 803L. For reasons that will become further apparent, in certain embodiments of the present invention, the left mouse device 803L should only be moved, but not clicked or scrolled in the system 800. Therefore, the representation of the second visible object 806 in the form of an arrow are confusing to a user, and a line or a rectangle or a square may be preferred as a visible indication of the second position, because it is intuitively clear that a line or square should not be clicked. In the system 800, the native mouse cursor 805 tracks movements of the right mouse device 803R, and is therefore probably better suited for right-handed people.
FIG. 9 shows a computer system 900 according to an embodiment of the present invention which can be viewed as a variation of the computer system 800 shown in FIG. 8,
wherein the native mouse cursor 905 tracks movements of the left mouse device 903L. The embodiment of FIG. 9 is therefore probably better suited for left-handed people. In this embodiment, the black arrow 906 can be moved by the right pointing device 903R, but this pointing device 903R should not be clicked or scrolled in certain embodiments of the present invention.
FIG. 9 (a) shows that the position of the black arrow 906 is adjusted in accordance with movements of the right pointing device 903R.
FIG. 9 (b} shows that the position of the native mouse cursor 905 is adjusted in accordance with movements of the left pointing device 903L. Both arrows are visible on the screen at all times.
FIG. 9 {c) shows an exemplary position update routine as part of a message processing routine, in pseudo code, that can be used to achieve this behavior. As can be seen, the first and second position Pos1, Pos2 are updated in accordance with movements of the first and second pointing devices 903R, 903L, respectively. In the event that the left pointing device 903L was moved, the position of the native mouse cursor 905 is adjusted accordingly. In the event that the right pointing device 903R was moved, the position of the native mouse cursor 905 is forced back to the second position Pos2 maintained by the overlay application. Thus, the two arrows 905, 906 are individually movable by the two pointing devices.
FIG. 10 shows a computer system 1000 according to another embodiment of the present invention, which can be viewed as a variation of the computer system 700 of FIG. 7 for aspects related to the positioning of the native mouse cursor 1005, which can be seen as a variant of the computer system 800 of FIG. 8 and 900 of FIG. 9 for aspects related to the shape of the second visible object 1006.
The computer system 1000 includes a computing device 1001 connected to a first pointing device 1003R and to a second pointing device 1003L and connected to a display screen 1004. The computing device 1001 has at least one processor that performs an O / S and GUI that has a native mouse pointer / cursor 1005. (e.g., a white arrow), and performs an overlay application according to an embodiment of the present invention. The overlay application maintains a first and a second pointing position Pos1, Pos2 in accordance with movements of the first and second pointing device 1003R, 1003L, respectively, and adjusts the position of the native mouse pointer / cursor 1005, e.g. overwrites it, in accordance with movements of the pointing device that was most recently moved.
FIG. 10 (a) shows movements of the first visible object, being the native mouse pointer 1005, in accordance with movements of the first pointing device 1003R because this is the pointing device being moved, while the second visible object 1006 (e.g., a black arrow) shows the other maintains position Pos2. In this situation, the user may press / release / click the buttons and / or move the scroll wheel of the first mouse pointer 1003R, for example, to activate the text editor application, select text, add comments, etc. FIG. 10 {b} shows movements of the first visible object 1005 in accordance with movements of the second pointing device 1003L, which is the pointing device being moved, while the second visible object 1006 (e.g. a black arrow!) Assumes the other maintained position Pos1 . In this situation, the user can press / release / click and / or move the scroll wheel of the second mouse pointer 1003L, for example, to activate the audio player application 1017, click the play or pause button, etc.
As can be understood from FIG. 10 (a) and FIG. 10 (b), the user can see both locations on the screen 1004, and can work in two different locations on the screen, albeit not simultaneously, simply by moving the respective pointing device. The user can click on any mouse device or rotate the appropriate mouse wheel after ensuring that the native mouse cursor 1005 is in the desired maintained position Pos1 or Pos2, as indicated visually by the white arrow.
FIG. 11 shows a specific example of the simplified high-level block diagram of hardware and software components shown in FIG. 5 (f) in the event that the pointing devices M1, M2 are HID compatible devices (abbreviation for the "Human Interface Device"), and in the event that the computer device includes an HID interface with at least two connection ports, such as typically the the case is, inter alia, for computer devices containing USB interfaces. As known in the art, such mouse devices send signals or messages to the computer device, e.g. in the form of HID packets, containing movement or movement information dx, dy and / or information about a button that is pressed or released, and / or information about a scroll wheel being spun. In fact, the communication between the computer device and the mouse devices is bi-directional, but the overlay application does not need to handle that portion of the communication, which is therefore not discussed further.
As schematically indicated by the asterisk *, only the overlay application 1151 is new in FIG. 11, all other components (without asterisk) can be classical components, in particular the pointing devices M1, M2, the operating system O / S and GUI 1160, the mouse device drivers 1161, and the other applications 1162. This again illustrates that the overlay application can be used to extend or enhance the functionality of an existing O / S and GUI that originally provides only a single native mouse cursor.
The mouse devices M1 and M2 can be wired mouse devices, or wireless mouse devices, or a combination of both.
FIG. 12 shows a variation of the high level block diagram of FIG. 11, wherein two classic pointing devices M1, M2 are connected to the computing device via a classic hub or adapter or dongle 1220 instead of being directly connected to the computing device.
The hub 1220 can be a wired hub or a wireless hub.
The interface can be an HID compatible interface, and the pointing devices can be HID compatible devices, but this is not absolutely required for the present invention to work.
FIG. 13 shows some lines of pseudo-code illustrating a portion of a message processor capable of operating in one of three input modes, referred to herein as follows: - "Two Handed" (FIG. 13 (a)) as explained in FIG. . 7 (c), wherein the native mouse cursor follows movements of the pointing device most recently moved, or - "" Right Handed "(FIG. 13 (b)) as explained in FIG. 5 (g), where the native mouse cursor follows movements of the right pointing device, and - "Left Handed" (FIG. 13 (c)) as explained in FIG. 9 (c), where the native mouse cursor follows movements of the left pointing device.
Some embodiments of an overlay application according to the present invention may implement only one of these input modes (e.g., hard coding). Other embodiments of an overlay application according to the present invention can potentially implement two of these input modes, or all three of these input modes, and allow the user to select which input mode to use, for example explicitly, as shown in FIG. 31 (a), by allowing a user to select a "Dominant mouse", in which case, for example, the display modes of groups 2850 and 2870 (see e.g.
FIG. 28) can be hidden, or implicitly by allowing the user to select a display mode from any group 2850 to 2870, in which case the application can change the corresponding input mode itself, if necessary.
FIG. 14 (a) is a schematic representation of an exemplary sequence of signals or messages from two classic mouse devices M1, M2, showing how these signals are properly handled by the message processing process shown in FIG. 13, which is assumed to be configured in the "Ambidextrous" input mode. In reality, many more signals or messages can be exchanged between the computer device and the mouse devices, but these are not shown in order not to overload the drawings.
Also, FIG. 14 only how the position of the native mouse cursor NC is updated, and does not explicitly mention how the position of the second and / or further visible object is updated, again in order not to overload the drawings.
At time t1, mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response, the overlay application updates Pos2 and sets the position of the native mouse cursor NC to Pos2. Depending on where the native mouse cursor was located before t1, this may appear to the user as a small movement or a jump.
At time t2, mouse device M1 is moved and sends a motion message mov1 to the computer device, and in response, the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Pos1, which will typically appear to the user as a "jump" from Pos2 to Pos1.
At time t3, mouse device M1 is moved again and sends a motion message mov1 to the computer device, and in response, the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Pos1, which will typically occur to the user as a little movement.
At time t4, mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response the overlay application updates Pos2 and sets the position of the native mouse cursor NC to Pos2, which will typically appear to the user as a "jump" from Pos1 to Pos2.
At time t5, mouse device M1 is moved and sends a motion message mov1 to the computer device, and in response, the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Pos1, which will typically appear to the user as a "jump" from Pos2 to Pos1.
FIG. 14 {b} shows the same sequence of signals or messages as that of FIG. 14 (a), showing how an overlay application that uses the message processor of FIG. 13, when configured in "Right-handed" input mode, would update the position NC of the native mouse cursor. As can be seen, when the right pointing device M1 is moved, the first position Pos1 is updated, and when the second pointing device M2 is moved, the second position Pos2 is updated, but in all these cases the position of the native mouse cursor NC is set at the first position Pos1, associated with movements of the right pointing device MI.
FIG. 14 (c) shows the same sequence of signals or messages as that of FIG. 14 (a), showing how an overlay application that uses the message processor of FIG. 13, when configured in the "Left-handed" input mode, would update the position NC of the native mouse cursor. As can be seen, when the right pointing device M1 is moved, the first position Pos1 is updated, and when the second pointing device M2 is moved, the second position Pos2 is updated, but in all these cases the position of the native mouse cursor NC is set at the second position Pos2, associated with movements of the left indicating device M2.
As can be understood from FIG. 14 (a) to FIG. 14 (c), movements of both mouse devices M1, M2 are handled correctly, even simultaneous movements of the two mouse devices M1, M2.
FIG. 15 is a schematic representation of another exemplary sequence of signals or messages from two classic mouse devices M1, M2, showing how these signals are properly handled by the message processing process shown in FIG. 13, which is assumed to be configured in the input mode. " Two-handed".
At time t1, mouse device M1 is moved and sends a motion message mov1 to the computer device, and in response, the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Pos1. Depending on where the native mouse cursor NC was located before t1, this may appear to the user as a small movement or a jump.
At time t2, mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response the overlay application updates Pos2 and sets the position of the native mouse cursor NC to Pos2, which will typically appear to the user as a jump from Pos1 to Pos2.
At time t3, a button (e.g. a left button M2B1) of the second mouse device M2 is pressed, and the mouse device M2 sends a button-pressed message "press2" to the computer device. The message processor of the overlay application receives the message, but since it is not a motion message, step h) of FIG. 5 (e) is skipped, and the overlay application does not update Pos1 or Pos2, but leaves the O / S and the GUI and / or the respective underlying application pressing the button M2B1 handle.
At time t4, the button M2B1 of the second mouse device M2 is released and the second mouse device M2 sends a push-button release message "release2" to the computer device, and again the overlay application does not change a position, but leaves the O / S and the GUI and / or the relevant underlying application handle the release of the button. As is known in the art, the O / S will, under some conditions, consider pressing a button and releasing a button as a so-called "mouse click".
At time t5, mouse device M1 is moved and sends a motion message mov1 to the computer device, and in response, the overlay application updates Pos1 and sets the position of the native mouse cursor NC to Pos1, which will typically appear to the user as a jump from Pos2 to Pos1.
As can be understood from FIG. 15, the "click" on a button M2B1 of the second mouse M2 is correctly handled by the message processor of FIG. 13, configured in Two-Handed input mode, despite movements of the first mouse M1 before and after the click.
In a variation of FIG. 15 (not shown), the mouse M2 is moved after pressing the button M2B1 at time t3 and before releasing the button M2B1 at time t4. In this case, the position update routine will update the position of the native mouse cursor, which has the effect of dragging (dragging). Drag also works well with the message processor of FIG. 13.
However, tests with further sequences have shown that the message processor of FIG. 13 does not always provide the expected behavior. Further research has shown that ao. the exemplary sequence of signals or messages shown in FIG. 16 is treated unsatisfactorily.
At time t1, mouse M1 is moved and sends a motion message mov1 to the computer device, and in response the message processor updates the first position Pos1 and positions the native mouse cursor accordingly. Likewise at time t2.
At time t3, a button (e.g., a left button M1B1) of the first mouse device M1 is pressed and the mouse sends a button-pressed message press1 to the computer device, for which the message processor of FIG. 13 does not take any action.
At time t4, the second mouse device M2 is moved and sends a motion message mov2 to the computer device, and in response, the message processor of FIG. 13 the native mouse cursor NC to the second maintained position Pos2.
At time t5, the button M1B1 of the first mouse device M1 is released and the mouse device M1 sends a push-button released message release to the computer device, but the O / S considers that this release occurs on Pos2, which is usually different from Pos1, which does not result in the expected "click" behavior of the first mouse M1, but in a "drag" behavior. This is not the intended behavior.
In a first attempt to solve this problem, the message processor of FIG. 13 modified by adding and maintaining two variables, a boolean variable M1pressed (M1 pressed) to indicate whether a button of the first mouse device M1 is pressed or released, and a variable M2pressed (M2 pressed) to indicate whether a button of the second mouse device M2 is pressed or released, and by not responding to the motion signal mov2 sent by the second mouse M2 at t4 during the time that a button of the first mouse M1 is pressed, as shown in FIG. 17, but for some reason this changed message processor did not provide the correct result either.
In a second attempt to solve this problem, the message processor of FIG. 13 further changed, but instead of ignoring the "mov2" signal (as in FIG. 17), the message processor was changed to actively reposition the native mouse cursor NC at the same position Pos1 as where the native mouse cursor was at time t3 when the button was pressed. Tests have shown that such a message processor works for a "click" on a button of the first mouse M1, but does not work when the first mouse M1 is "dragged". Instead of freezing (at t4) the Pos1 position at the same position where the native mouse cursor NC was located when the button was pressed (at t3), the position of the native mouse cursor NC was allowed to move along with movements of the first mouse M1, even when a button of the same device M1 was pressed.
FIG. 18 (a) shows some lines of pseudo code illustrating part of an improved message processing routine to implement this behavior. (only part of Two-Handed input mode is shown).
FIG. 18 (b) shows a variation of the routine of FIG. 18 (a), as can be used in overlay applications according to embodiments of the present invention. (again only part of the Two-Handed input mode is shown). The main differences from the routine of FIG. 18 (a) are that the second position Pos2 is updated even if a button of the first pointing device is pressed, and vice versa, the first position Pos1 is updated even if a button of the second pointing device is pressed. This routine will be discussed further when discussing FIG. 26 (a) and FIG. 26 (b).
FIG. 18 (c) depicts a more complete version of a position updating routine, as can be used in overlay applications according to embodiments of the present invention. This can be seen as a combination of the code portions shown in FIG. 5 (g) and in FIG. 9 (c) and in FIG. 18 (a).
FIG. 19 shows the same exemplary sequence of signals or messages of FIG. 16, but processed by the message processor of FIG. 18 (a). As can be seen, in response to the motion signal mov2 from the second mouse M2 at time t4, the improved message processor of FIG. 18 (a) the second position Pos2 at in accordance with the motion signal mov2, but actively repositions the native mouse cursor to Pos1, so that at time t5, the button M1B1 of the first mouse device M1 is considered to be released at the same position Pos1 as where it was was pressed. The O / S interprets this as a "click" of the button M1B1 of the mouse device M1, which has been handled satisfactorily, despite the movement signal "Mov2" from the second mouse M2 at time t4.
It is noted that updating Pos2 at t4 is not required for the correct interpretation of the click command sent by the first pointing device M1, and can optionally be omitted. However, if omitted, the visible object (s) associated with the second position Pos2 will not move as long as a button of the first pointing device M1 is pressed, even if the second pointing device is moved. To allow for such movements, the position Pos2 is preferably updated even if a button of the first pointing device is pressed. FIG. 20 shows a variation of the sequence of signals or messages of FIG. 19, showing that dragging one mouse M1 works correctly even when the second mouse M2 is moved.
At time t1, the first mouse M1 is moved, the first mouse M1 sends a first motion message mov1 to the computing device, and in response the overlay application updates Pos1 and positions the native mouse cursor NC on Pos1.
At time t2, a button (e.g. a left button M1B1) of the first mouse M1 is pressed, the first mouse M1 sends a button-pressed message press1 to the computing device, and in response the message processor of the overlay application sets an internal variable M1pressed to TRUE.
At time t3, the second mouse M2 is moved, the second mouse M2 sends a second motion message mov2 to the computer device, and in response, the message processor updates Pos2 and positions the native mouse cursor NC on Pos1.
At time t4, the first mouse M1 is moved, the first mouse M1 sends a first motion message mov1 to the computing device, and in response the overlay application updates Pos1 and positions the native mouse cursor NC on Pos1.
At time t5, the button M1B1 of the first mouse device M1 is released, causing the O / S and GUI to perform a "drag" operation, and in response the message processor sets the variable M1pressed to FALSE.
As can be understood from FIG. 20, the dragging of the first mouse M1 is handled satisfactorily by the message processor shown in FIG. 18 (a) configured in Two-handed input mode, even in combination with movements of the second mouse M2.
Referring to FIG. 21, further tests were performed using the message processor of FIG. 18 (a) in the Two Handed input mode, which revealed yet another error condition, represented by the exemplary sequence of signals or messages shown in FIG. 21, where: At time t1, the second mouse M2 is moved, a move message mov2 is sent, the second position Pos2 is updated taking into account the displacement values dx, dy, and the native mouse cursor is set to Pos2.
At time t2, the first mouse M1 is moved, a motion message mov1 is sent, the first position Pos1 is updated taking into account the displacement values dx, dy, and the native mouse cursor is set to Pos1.
At time t3, the first mouse M1 is moved again, another move message mov1 is sent, the first position Pos1 is updated, and the native mouse cursor NC is set to the updated position Pos1.
At time t4, a button (e.g., a left button M2B1) of the second mouse device M2 is pressed, a button-pressed message press2 is sent, and in response, the message processor of FIG. 18 (a) Set the value M2pressed to TRUE, and change the position of the native mouse cursor NC to Pos2.
At time t5, the button M2B1 of the second mouse device M2 is released, a release message release2 is sent to the computing device, and in response the message processor sets the variable M2pressed to FALSE.
Despite the fact that a button of the second mouse device M2 was pressed and released at the same physical position, the press and release messages of FIG. 21 strangely enough not usually interpreted as a click. Although the inventors do not wish to be bound by theory, analysis seems to indicate that this sequence goes wrong, because at the time t4 when the button M2B1 of the second mouse M2 is pressed, the native mouse position NC is still at the first position Pos1 (x1, y1), and therefore the O / S and the GUI will assume that the "button press" occurs at the position Pos1 (x1, y1) instead of Pos2, and only some time later (than t4) the overlay application will reposition the native mouse cursor NC at the second position Pos2 (x2, y2), but that is "too late". This problem is not known in the art, because only one cursor position exists.
No technical solution to this problem seems possible because it is impossible for the overlay application to predict when a mouse button M2 will be pressed, and because the O / S and GUI will always be faster than the overlay- application. Desiring to leave the mouse driver layer 561, 1161, 1261 of the O / S and GUI intact, the inventors surprisingly came up with a different solution to this problem.
Referring to FIG. 22 (a), the inventors decided to provide a special mouse device M2 * that, when a button thereof is pressed, sends a "dummy movement" signal some time AT before sending an actual "button-pressed" signal. This effectively circumvents the problem of FIG. 21, because, as can be understood from FIG. 22 (a): At time t1, the second mouse M2 * is moved, a motion (real) message mov2 is sent to the computer device, in response the overlay application updates the second position Pos2 and sets the native mouse cursor NC to position Pos2 .
At time t2, the first mouse M1 * is moved, a motion (real) message mov1 is sent to the computing device, in response the overlay application updates the first position Pos1 and sets the native mouse cursor NC to position Pos1.
At time t3, a button (e.g. a left button M2B1) of the special second mouse device M2 * is pressed, causing the mouse device M2 * to first send a dummy motion message "dmov2" (preferably with small displacement values dx, dy) , and some time later AT sends a button-down message press2. As shown, in response to the dummy motion signal dmov2, the overlay application will set the native mouse cursor NC to the second position Pos2 well before the button-pressed message press2 is sent by the mouse device M2 * at time t3 + AT, such that the O / S and the GUI will assume that the "button press" occurs at position Pos2 (or very close to the original position Pos2, if dx and dy are small, but not exactly zero).
At time t4, the button M2B1 is released, and in response, the overlay application's message processor sets a variable M2pressed to FALSE.
It is noted that a dummy motion message is not absolutely necessary when the button is released, but only when a button is pressed, at least in this example. In practice, however, it is safer to send a dummy motion message even when released, in case the other pointing device is moving in the meantime.
FIG. 22 {b} shows a variant of the sequence of FIG. 22 (a) in which a scroll wheel of the second pointing device M2 * is rotated instead of pressing and releasing a button. In a manner similar to that described in FIG. 22 (a), the special pointing device M2 * does not merely send a scroll wheel rotation message, but first transmits a dummy motion message dmov2, and only some time AT later does the pointing device M2 * transmit the actual scroll wheel rotation message rotate2, in order to be possible. cause the overlay application to reposition the native mouse cursor NC before the actual scroll wheel rotation message arrives at the computer device.
In FIG. 22 (b) a scroll wheel of the pointing device M2 * is physically rotated at time t3, when the pointing device M2 * sends a dummy motion message dmov2 to the computer device. In response, the overlay application (configured in Two-Handed input mode) will update the second position Pos2 and set the native mouse cursor to Pos2.
At time t3 + AT, the pointing device M2 * sends a scroll wheel rotate message rotate2, which the overlay application can simply ignore, because this message will be handled by the O / S and the GUI.
While the solution presented in FIG. 22 (a) and FIG. 22 (b) requires a special mouse device, it is an advantage that this special mouse behavior does not require a hardware modification, but only a firmware modification of the mouse device.
It is also an advantage that the proposed solution is retro-compatible, in the sense that this "special mouse" with the "move-before-click" function and / or with the "move-advance-scroll" function can also be used in classic computer systems, which do not have a special overlay application.
The main difference from existing mouse devices is that the button-pressed signal and / or the scroll wheel spin signal is slightly delayed, which is probably totally unacceptable in the gaming world, and for that reason alone is not evident, but causes this no significant problem for professional users such as translators, proofreaders, transcribers, technical writers, lawyers, etc. who are used to working with tools (eg.
CAT tools, local or online dictionaries, local or online databases, local or online translation memories, etc.) that have fairly slow response times anyway (typically on the order of at least 500 ms). There is some design freedom in the choice of the three parameters AT, dx, dy.
In one embodiment, the delay AT is a value in the range of 1 ms to 250 ms or from 2 ms to 100 ms, and dx is a value in the range of -5 to +5, and dy is a value in the range of - 5 to +5. In one embodiment, the delay AT is a value in the range of 1 ms to 250 ms or from 2 ms to 100 ms, and dx is a value in the range of -2 to +2, and dy is a value in the range of - 2 to +2. In one embodiment, the delay AT is a value in the range of 1 ms to 250 ms or from 2 ms to 100 ms, and dx is a value in the range of -1 to +1, and dy is a value in the range of - 1 to +1. In one embodiment, the delay AT is a value ranging from 1 to 250 ms or from 2 ms to 100 ms, and dx = -1 and dy = -1. In one embodiment, the delay AT is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = -1 and dy = 0. In one embodiment, the delay AT is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = -1 and dy = +1. In one embodiment, the delay ΔT is a value in the range of 1 to 250 ms or 2 ms to 100 ms, and dx = 0 and dy = -1. In one embodiment, the delay AT is a value in the range of 1 to 250 ms or from 2 ms to 100 ms, and dx = 0 and dy = 0. In one embodiment, the delay AT is a value in the range of 1 to 250 ms or 2 ms to 100 ms, and dx = 0 and dy = +1. In one embodiment, the delay AT is a value in the range of 1 to 250 ms or 2 ms to 100 ms, and dx = +1 and dy = -1. In one embodiment, the delay AT is a value in the range of 1 to 250 ms or 2 ms to 100 ms, and dx = +1 and dy = 0.
In one embodiment, the delay AT is a value in the range of 1 to 250 ms or 2 ms to 100 ms, and dx = + 1 and dy = + 1.
For example, in all of the above embodiments, the value of AT may be at least 1 ms, or at least 2 ms or at least 8 ms, or at least 16 ms, or at least 24 ms, or at least 32 ms, or at least at least 40 ms, or at least 48 ms, or at least 56 ms, or at least 64 ms, or at least 72 ms, or at least 80 ms, or at least 88 ms, or at least 96 ms, or at least 104 ms, or at least 120 ms, or at least 144 ms, or at least 160 ms, or at least 176 ms, or at least 192 ms.
For example, in all of the above embodiments, the value of AT may be equal to about 8 ms or about 16 ms, or about 24 ms, or about 32 ms, or about 40 ms, or about 48 ms, or about 56 ms, or about 64 ms, or about 72 ms, or about 80 ms, or about 88 ms, or about 96 ms, or about 104 ms, or about 112 ms, or about 120 ms, or about 128 ms, or about 136 ms, or about 144 ms, or about 152 ms, or about 160 ms, or about 168 ms, or about 176 ms, or about 184 ms, or about 192 ms, or about 200 ms, or about 208 ms, or about 216 ms, or about 224 ms, or about 232 ms, or about 240 ms, or about 248 ms.
On the one hand, the value of AT should be large enough to increase the likelihood that the computing device will have sufficient time for the overlay application to process the dummy motion signal before the button-pressed signal arrives. On the other hand, the value of AT should be as small as possible to reduce the delay, and to reduce the risk that a real motion signal or a dummy motion signal from the other mouse device could get in between. The skilled person can find a suitable compromise.
In one embodiment, the dummy motion signal or message contains the displacement values dx = 0 and dy = 0. If this message is not filtered out by the O / S and GUI, such a message would be ideal, as it does not match the coordinates of the first and second position Pos1 or Pos2, but still allows the overlay application to do so. signal to reposition the native mouse cursor (or in other words, it allows the overlay application to force a jump if necessary). If the dummy motion message has a zero displacement (dx = 0 and dy = 0), the native mouse cursor NC will not shift or drift due to multiple dummy motion messages being sent no matter how many times the button is clicked, and / or scroll wheel is scrolled.
In another embodiment, the dummy motion message is not a single predetermined message with dx = 0 and dy = 0, but consecutive dummy motion messages are different from the first dummy motion message, with dx and dy values selected such that a cumulative horizontal displacement (e.g. due to multiple presses and releases) defined as Ax = 5> dx and a cumulative vertical displacement defined as Ay => dy are less than a predetermined maximum distance. Preferably, the predetermined maximum distance is less than 10 pixels, more preferably less than 4 pixels, or less than 3 pixels, or less than 2 pixels.
For example, the dummy motion messages may be alternately or periodically selected from a group consisting of two predetermined dummy motion messages, including a first message with (dx = + 1, dy = 0) and a second message with (dx = -1, dy = 0), which leads to a small left / right movement, but other groups of two messages are also possible, for example: - the group consisting of two dummy movement messages containing a first message with (dx = 0, dy = + 1) and a second message with (dx = 0, dy = -1), leading to a small up / down movement, or - the group consisting of two dummy motion messages containing a first message with (dx = + 1, dy = + 1) and a second message with (dx = -1, dy = -1), leading to a small diagonal movement, or - the group consisting of two dummy motion messages containing a first message with (dx = + 1, dy = -1) and a second message with (dx = -1, dy = + 1), leading to a small diagonal movement, or - the group consisting of three dum my motion messages containing a first message with (dx = + 1, dy = 0), and a second message with (dx = 0, dy = + 1), and a third message (dx = -1, dy = -1) , or - the group consisting of four dummy motion messages, containing a first message with (dx = + 1, dy = 0), a second message with (dx = 0, dy = -1), a third message with (dx = -1, dy = 0), and a fourth message with (dx = 0, dy = + 1), etc.
In this way, the mouse position remains substantially stationary, even after multiple clicks and / or scroll wheel rotations, and does not drift away from its original position unless the pointing device is physically moved.
However, a cumulative zero displacement can also be achieved in other ways, e.g. by sending a first dummy motion message with a predetermined displacement when a button is pressed, (e.g. with dx = + 2, dy = -1) , and when the button is released, send a second dummy motion message with an opposite displacement (e.g., with dx = -2, dy = + 1).
FIG. 23 (a) shows a simplified high-level block diagram of hardware and software components contained in an exemplary computer system according to an embodiment of the present invention, including a computer device 2301 with a special overlay application 2351 according to an embodiment of the present invention, and two special mouse devices M1 *, M2 * according to an embodiment of the present invention. The special mouse devices M1 *, M2 * have the aforementioned "move-before-click" function shown in FIG. 22 (a) and / or the aforementioned "move-before-scroll" function shown in FIG. 22 (b), which means that these mouse devices M1 *, M2 *, upon detection of a button being pressed and / or a scroll wheel being rotated, first send a "dummy motion message" and only after some time AT later a button- press message or send a scroll wheel spin message.
As suggested by the asterisks, only the overlay application 2351 and mouse devices M1 *, M2 * are special. All other components, in particular the O / S and the GUI 2360, the device drivers 2361, the other applications 2362, etc. can be classic components.
The mouse devices M1 * and M2 * can be wired mouse devices, or wireless mouse devices, or a combination of both.
But the present invention is not limited to special mouse devices, and the proposed solution can also be applied to other pointing devices that have at least one button and / or at least one scroll wheel.
FIG. 23 (b) shows an exemplary block diagram of a special pointing device according to an embodiment of the present invention, as may be used in the system of FIG. 23 (a) or FIG. 24 (which will be discussed below). The pointing device 2303 includes at least: a motion detecting mechanism 2321 for detecting a two-dimensional motion; - a control unit 2324 connected to said motion detecting mechanism 2321; - at least one button B1 depressible and releasable by a user; - a button detecting mechanism 2322 for detecting whether said at least one button B1 is pressed or released; wherein the control unit 2324 is connected to the button detecting mechanism 2322; wherein the control unit 2324 is adapted for (e.g. programmed for): * when a motion is detected, to send a motion message mov2; * when it is detected that the at least one button B1 is pressed, for sending a dummy motion message dmov2, and for sending a button-pressed message press2 a predetermined period AT after sending the dummy motion message dmov .
The motion message may include displacement parameters dx, dy related to the two-dimensional motion.
The pointing device 2303 may further include at least one scroll wheel W1, rotatable by a user, and a scroll wheel detecting mechanism 2323 for detecting whether it has rotated at least one scroll wheel W1, and in which direction (e.g., up or down), and the controller 2324 may be connected to the scroll wheel detecting mechanism 2323, and may be further adapted, when it is detected that it has rotated at least one scroll wheel, to send a dummy
motion message dmov, and for sending a scroll wheel spin message a predetermined period AT after sending the dummy motion message dmov. The pointing device 2303 may further comprise an RF transceiver 2325 connected to the control unit 2324, and the control unit 2324 may be further configured to send messages to a computer device 2301, and to receive messages from the computer device 2301 via said RF transceiver 2325, for example connected via a dongle 2311 to the computer device 2301. The control unit 2324 may be a programmable control unit, e.g. a microprocessor.
FIG. 24 shows a variation of the high level block diagram of FIG. 23 (a), wherein the two "special" pointing devices M1 *, M2 * are connected to the computing device 2401 through a classic hub or adapter or dongle 2420 instead of being directly connected to the computing device. The hub 2420 can be a wired hub or a wireless hub. The main purpose of this figure is to demonstrate that the mouse devices do not need to be directly connected to the computer. FIG. 25 shows a variation of the high level block diagram of FIG. 24, wherein the FIG. 21, the problem shown is solved in a dedicated hub 2520 rather than in dedicated pointing devices. Thus, the computer device 2501 of FIG. 25 preferably includes a classic O / S and GUI 2560, classic applications 2562, classic device drivers 2561, e.g. mouse drivers, classic pointing devices M1, M2, but a special overlay application 2551 and a special hub or adapter or dongle 2520. The functionality of the dedicated hub 2520 will be explained in more detail in FIG. 26 (a) to FIG. 26 {d}. Exemplary hardware block diagrams to implement this functionality will be described in FIG. 27 (a) and FIG. 27 (b). As can be understood from FIG. 21 to FIG. 22 (b), the hub H * should prevent the computing device from receiving a button-pressed message from pointing device M2 which is preceded in time by a message from the other pointing device M1 (e.g. a motion message, a button-click message, a push button released message, a scroll wheel message), and vice versa. Since the messages from both pointing devices must pass through the hub H *, the dedicated hub is able to avoid this situation. In the exemplary sequence of FIG. 26 (a), this situation is avoided as follows. At time t1, the first pointing device M1 sends a motion message mov1, and the hub H * sends the motion message to the computer device (also called "host").
At time t2, a button M2B1 of the second pointing device M2 is pressed, and the pointing device M2 sends a button pressed message to the hub H *. The hub H * intercepts this message, and sends a motion dummy message dmov2 to the host (preferably with small motion parameters as explained above), preferably formatted as if this motion dummy message were sent by the second pointing device M2.
Some time later, at time t2 + AT, the hub 2520 sends the button-pressed message to the host, packaged or formatted as if it were from the second pointing device M2.
In the embodiment shown in FIG. 26a, the hub H * ignores all messages sent by the other pointing device M1 until the button M2B1 of the pointing device M2 is released at time t5. Therefore, in the example shown, the motion message mov1 sent by the pointing device M1 at t3 is ignored.
At time t4, the second pointing device M2 sends a push-button released message to the hub H *, and the hub H * sends this message to the host.
At time t5, the first pointing device M1 sends a motion message to the hub H *, and the hub H * sends this message to the host.
In other words, the special hub H * that shows the behavior of FIG. 26 (a) implements all messages coming from the pointing devices to the host, except when a button-pressed message is received from one of the pointing devices (in the example M2), in which case the hub H * is temporarily on a special behavior, namely by first sending a dummy motion message, and some time later AT sending a button-pressed message press2, and by ignoring all messages from the other pointing device (in the example: M1) to a push button - released message is received from the pointing device M2 whose button M2B1 was pressed.
As explained above for the special pointing device, (but not shown in FIG. 26a) it is also possible here to also send a dummy motion message at t4 when the button M2B1 is released, and only send the actual button released message. at t4 + AT, e.g. in case the dummy movement is not zero displacement.
FIG. 26 (b) shows the same sequence as FIG. 26 (a), but treated differently by a variant of the special hub H *. In this embodiment, the hub does not ignore the motion message mov1 sent on t3, but delays this message until after the button-pressed press2 message is sent to the host on t2 + AT, in the example at tx = t2 + AT + e, where € preferably less than 33 ms, for example about 8 ms or about 16 ms or about 24 ms. As explained above, the dedicated overlay application can optionally update Pos1, but takes some time to reposition the native mouse cursor on Pos2, otherwise the O / S may interpret the button as released on Posl instead of Pos2. Therefore, in this example, the push-button released message sent by the second pointing device M2 at t4 must also be delayed to tx + AT. Obviously, if t4 was already greater than tx, then the push-button released message release2 can easily be forwarded to the host.
If the behavior of the signals and how the overlay application will respond to them, shown in FIG. 26 (b) is well understood, it can now also be understood that the position update routine of FIG. 18 (b), which allows the second object to be moved while a button of the first pointing device is pressed, can go wrong for the sequence shown in FIG. 26 (b), in the event that the user releases the button M2B1 before tx + AT. The shown in FIG. 18 (a) routine shown does not suffer from this problem because it ignores the motion signal, in much the same way as the hub of FIG. 26 (a) does this.
Thus, when the dedicated hub H * (having the behavior as explained with reference to FIG.26a or FIG.26b) is used, in the overlay application both the position update routine of FIG. 18 (a) as well as that of FIG. 18 (b) both are used in Two Handy input mode, because the hub provides the timing between the messages coming from different devices. But if the dedicated hub is not used in the computer system, then the position update routine shown in FIG. 18 (a) "safer".
FIG. 26 (c) shows an exemplary sequence to explain how the dedicated hub H * depicted in FIG. 26 (a) has specified behavior can respond in response to a mouse wheel message.
At time t1, the first pointing device M1 sends a motion message mov1, and the hub H * sends the motion message to the computer device (also called "host").
At time t2, a scroll wheel M2W1 of the second pointing device M2 is rotated, and the pointing device M2 sends a scroll wheel rotation message to the hub H *. The hub H * intercepts this message, and sends a dummy motion message dmov2 to the host (preferably with small motion parameters as explained above), preferably packaged or formatted as if this dummy motion message were sent by the second pointing device M2.
Sometime later, at time t2 + AT, the hub H * sends the scroll wheel rotation message to the host, packaged or formatted as if it came from the second pointing device M2.
In the embodiment shown in FIG. 27, the hub H * ignores all messages sent by the other pointing device M1 to the hub H * arriving between t2 and t2 + AT. Therefore, in the example shown, the motion message mov1 sent by the pointing device M1 at t3 is ignored.
At time t4, the first pointing device M1 sends a motion message mov1 to the hub H *, and the hub H * sends this message to the host.
At time t5, the second pointing device M2 sends a motion message to the hub H *, and the hub H * sends this message to the host.
FIG. 26 {d} shows the same sequence as FIG. 26 (c), but treated differently by a variant of the special hub H *. In this embodiment, the hub H * does not ignore the motion message mov1 sent on t3, but delays this message until after the scroll wheel rotation message scroll2 is sent to the host on t2 + AT, say at tx = t2 + AT + e.
The movement message mov1 on t4 sent by the first pointing device M1, and the movement message mov2 sent by the second pointing device M2 are simply sent to the host.
When comparing the behavior of the dedicated hub H * as specified in FIG. 26 (a) to FIG. 26 (d) with the behavior of the special pointing devices, as specified in FIG. 22 (a) and FIG. 22 (b), it can be seen that the special hub H * solves the problem of FIG. 21 approaches based on the same principle, namely by inserting a dummy motion message AT some time before forwarding the actual button pressed message or scroll wheel spin message.
Implementing this behavior in a dedicated hub rather than dedicated pointing devices provides the additional benefit that the hub H * can actually prevent the other pointing device from sending an intermediate message to the host between this dummy motion message and the actual button-pressed or scroll wheel spin message, and therefore can guarantee that the situation of FIG. 21 cannot occur.
This is not possible with the dedicated pointing devices without the hub.
Or in other words, although the "move-before-click" and the "move-before-scroll" function shown in FIG. 22 reduces the risk that the computer device will misinterpret the messages, the risk is not completely eliminated, because one pointing device has no influence over the messages sent by the other pointing device.
In contrast, a special hub H * with the features described in FIG. 26 (a) to FIG. 26 (d) full control over the messages sent to the computing device, and can filter and / or adjust the flow of messages to and from both pointing devices.
In this way, unwanted combinations of conditions or unwanted timing between the signals / messages can be avoided or adjusted so that the O / S and the GUI and the special overlay application 2551 will function correctly. "Smart hubs" that interpret and / or modify and / or insert and / or delay messages from multiple pointing devices to allow dual mouse support on a computing device in general, let alone facilitate dual mouse support provided by an overlay device. application running on a computing device with an O / S and GUI that provides only a single native mouse cursor, to reposition the native mouse cursor, to provide an O / S and GUI that provides virtual dual mouse support, do not exist in the mode of technology.
FIG. 27 (a) shows an exemplary block diagram of a hub or plug or adapter according to an embodiment of the present invention, as may be used in the system of FIG. 25, which has at least two physical connectors CON1, CON2 for connection to two classic wired pointing devices M1, M2, and at least one connector CON3 (optionally at the end of a cable) for connection to a computer device 2701.
FIG. 27 (b) shows an exemplary block diagram of a wireless hub (or dongle or adapter) according to an embodiment of the present invention, as may be used in the system of FIG. 25, which has one or two RF transceivers for operable connection to two classic wireless pointing devices.
The hub 2720 of FIG. 27 (a) comprises: - a first interface CON1 for communicating with a first pointing device M1; - a second interface CON2 for communicating with a second pointing device M2; - a third interface CON3 for communicating with a computer device 2701; a control unit 2744 which is connected to the first interface for receiving first messages from the first pointing device M1, and which is connected to the second interface for receiving second messages from the second pointing device M2, and which is connected to the third interface for sending the first and second messages to the computer device (2701); and wherein the control unit 2744 is adapted to send a dummy motion message a predetermined period (AT) before sending a button pressed message and / or a scroll wheel spin message.
More specifically, the control unit 2744 can be adapted to behave as follows: * upon receipt of a first button press message press1 from the first pointing device via the first interface, to send a first dummy motion message dmovl to the computer device via the third interface, and to send the first button-pressed message press1 to the computer device via the third interface a predetermined period AT later; * upon receipt of a second button push message press2 from the second pointing device via the second interface, to send a second dummy motion message dmov2 to the computer device via the third interface, and to a predetermined period AT later, the second button send pressed message press2 to the computing device via the third interface; * upon receipt of a first scroll wheel rotation message from the first pointing device via the first interface, to send a first dummy motion message dmov1 to the computer device via the third interface, and to a predetermined period AT later, the first scroll wheel- send rotation message to the computer device via the third interface;
* upon receipt of a second scroll wheel rotation message from the second pointing device via the second interface, to send a second dummy motion message dmov2 to the computer device via the third interface, and to a predetermined period AT later, the second scroll wheel- send a rotation message to the computing device via the third interface.
The hub 2721 of FIG. 27 (b) includes: - a first RF transceiver RFX1 for receiving messages from and sending messages to the first pointing device M1; and - a second RF transceiver RFX2 for receiving messages from and sending messages to the second pointing device M2, wherein the second RF transceiver is the same as the first RF transceiver, or is a separate RF transceiver; and - a connector CON3 connectable to a computer device 2701; a control unit 2745 connected to the first RF transceiver RFX1 for receiving first messages from the first pointing device M1, and connected to the second RF transceiver RFX2 for receiving second messages from the second pointing device M2, and connected to the connector CON3 for sending messages to (and typically also receiving messages from) the computer device 2701, the control unit 2745 being adapted to send a dummy motion message a predetermined period AT before sending a button-pressed message to the computer device and / or for sending a dummy motion message a predetermined period (AT) before sending a scroll wheel spin message to the computer device.
What has been described above with regard to the parameters dx, dy, AT when discussing special pointing devices is not repeated here, but is also applicable to the dedicated hub.
The control unit 2744, 2745 may be a programmable control unit, e.g., a microprocessor.
FIG. 28 (a) shows an exemplary user interface window, referred to herein as the "mini user interface", as may be used in embodiments of the present invention to configure the overlay application, e.g., to select an input mode and / or a display mode. choose, and / or choose a size and / or a color of the second or further visible object, and / or choose a semi-transparency level of the click-through window using one or more pop-up menus, an example of which is shown in FIG. 28 (b).
The user interface window, e.g., window 585 in FIG. 5c or window 885 in FIG. 8c, is preferably opaque and is not configured in click-through mode, and preferably has a Z-order higher than that of the click-through window 584, 884 in FIG. 5c and FIG. 8c. The pop-up menu of FIG. 28 (b) may appear, for example, upon clicking a right button of the dominant pointing device when the native mouse cursor is positioned over one of the icons representing a specific display mode. The overlay application may implement one or more input modes selected from the group consisting of: a One Hand Pointing mode (designated "One H" in FIG. 28a), a Dual Pointing mode where the left pointing device is the dominant pointing device (designated as LeftH "in FIG. 28a), a Dual Pointing mode where the right pointing device is the dominant pointing device (referred to as" RightH "in FIG. 28a), and a Dual Pointing mode in which both pointing devices are dominant (referred to as" TwoH "in FIG. 28a). In an embodiment of the present invention which is configurable in one of multiple input modes, a message processor of the overlay application may include the pseudo-code of FIG. 18 (c). It is noted that testing whether a button of the dominant pointing device is pressed or not is not required in the case of Right Handy and Left Handy input mode, because the native mouse cursor does not jump from the one to another maintained position Pos1, Pos2. So for example, in the case of the Right-handed input mode, a button on the right pointing device can be safely pressed and moved (e.g. when dragging or when selecting text in a word processor) while the left pointing device is also moved without a conflict. (assuming that no left pointing device button is pressed, and the left pointing device scroll wheel is not spinning).
It is noted in this regard that some embodiments of the dedicated hub have the behavior shown in FIG. 26 (a) and FIG. 26 (c), or as shown in FIG. 26 (b) and FIG. 26 (d), assuming that both pointing devices are allowed to be moved and / or clicked and / or scrolled.
Other embodiments of the dedicated hub can unconditionally ignore all button pressed messages and all push button released messages and all scroll wheel spin messages received through one of its input ports (e.g. the port where the non-dominant hand pointing device is connected) , and only allow motion messages through that input port. Such an embodiment may be particularly useful in conjunction with a Left Handy or Right Handy input mode. In this way, it is ensured that such message sent by the non-dominant pointing device cannot reach the computer device, even if the user were to accidentally click or scroll the non-dominant pointing device.
Some embodiments of the dedicated hub are configurable, e.g. via dip-switches or in software, e.g. via a command from the overlay application, or otherwise, to either (i) allow the passage of all motion and push and release and scroll wheel rotation messages from all pointing devices, or (ii) allow passage of motion messages from all pointing devices, but allow only push and release and scroll wheel messages from a single pointing device.
The overlay application may implement one or more display modes, for example, as represented by the icons shown in FIG. 28 (a). FIRST GROUP: A first group of 2840 display modes works with the "One Hand Pointed" input mode, and provides the following display modes, which can be summarized as follows: - icon 2840 (a) represents a mode with the single (native) mouse pointer provided by a classic O / S and GUI, - icons 2840 (b) to 2840 (f) represent a mode with the single (native) mouse pointer, but additionally a second and / or third visible object in the form of (b) a hand ; (c) a square or line segment or rectangle; (d) a line or rectangle that extends the full width of the overlay window; (e) a small cross; (f) a large cross formed by a horizontal line or rectangle preferably extending the full width of the overlay window 584, 884 and a vertical line or rectangle preferably extending the full height of the overlay window 584, 884; icon 2840 (j) representing a mode of a horizontal line or bar similar to mode 2840 (d), but with an additional visible object in the form of a small block, e.g. a square or rectangle movable within the line or bar; - icon 2840 (k) represents a mode with a vertical line splitting the screen into a left part and a right part, this vertical line is preferably movable by "dragging" (after temporarily setting the overlay window 584, 884 in "no-click-through mode" as explained in more detail in the co-pending "perforated bitmap application (s)", and with a horizontal line at the left portion of the vertical line, which horizontal line moves with the mouse pointer when the mouse pointer is on the left side of the vertical line, and which horizontal line freezes when the mouse pointer is on the right side of the vertical line.This mode can be especially useful for right-handed translators or reviewers who work with just a single mouse pointer; icon 2840 (l) represents a mode similar to mode 2840 (k) but the horizontal line is to the right of the vertical line, and move t up / down when the native mouse pointer is on the right side of the vertical line, and freezes when the mouse pointer is on the left side of the vertical line. This mode can be especially useful for left-handed translators or reviewers working with just a single mouse pointer.
In all modes 2840 (a) - (I), the native mouse cursor is visible on the screen, moving with the at least second visible object in the form of a hand or a line or a cross or a block as described above, except in the "frozen line" modes represented by pictogram 2840 (k) and 2840 (1), where the visible object moves with the pointing device only under certain conditions.
SECOND GROUP: A second group of 2850 display modes operates with the “Left Handed Dual Pointers” input mode, and provides the following display modes, which can be summarized as follows:
icon 2850 (a) represents a mode with the single (native) mouse pointer provided by a classic O / S and GUI, and additionally a second visible object in the form of an arrow, e.g. a black arrow.
The native mouse cursor is controlled by the left pointing device.
Aspects of mode 2850 (a) were also described in FIG. 9;
- icons 2850 (b) to 2850 (f) represent a mode with the single (native)
mouse pointer steerable by the left pointing device, and with an additional second or further visible object in the form of (b) a hand; (c) a square; (d) a line or rectangle; (e) a small cross; (f) a large cross, controllable by the other pointing device, namely the right pointing device;
- icon 2850 (g) to icon 2850 (i) represents a mode similar to mode 2850 (c) to
2850 (e), but the overlay application further shows an additional (g) square or (h) line or (i) small cross that moves with the native mouse pointer, and is therefore movable by the left pointing device;
- icon 2850 (j) represents a mode where the overlay application provides a vertical line splitting the screen into a left part and a right part, and with a first horizontal line located to the left of the vertical line moving in accordance with movements of the left pointing device, and with a second horizontal line located to the right of the vertical line moving in accordance with movements of the right pointing device.
The native mouse cursor is movable across the screen and moves with the left pointing device.
This mode can be especially useful for left-handed translators or reviewers comparing two texts;
- icon 2850 (!) represents a variant of mode 2850 (j) without showing the left horizontal line;
In all modes 2850 (a) - (l), the native mouse cursor is visible on the screen and moves in accordance with movements of the left pointing device, which is why these modes are probably best suited for left-handed people using the native mouse cursor. steer with their dominant (left) hand, and steer a second visible object with their non-dominant
(right) hand. The user should not click or scroll with the non-dominant pointing device, but only move it (e.g., to underline textual information on the screen). THIRD GROUP: A third group of 2860 display modes works with the "Right Handy Dual Pointing" input mode, and provides display modes similar to the second group 2850 of display modes, except that the functions of the left and right pointing devices are interchanged, and that in mode 2860 ( k) the left horizontal line is displayed in place of the right horizontal line.
Aspects of mode 2860 (a) were also described in FIG. 8.
Aspects of mode 2860 (d) were also described in FIG. 5.
Aspects of mode 2860 (h) were also described in FIG. 6.
In all modes 2860 (a) - ({k), the native mouse cursor is visible on the screen and moves in accordance with movements of the right pointing device, which is why these modes are probably best suited for right-handed people using the native mouse cursor with their dominant (right) hand, and move a second visible object with their non-dominant (left) hand. The user should not click or scroll with the non-dominant pointing device, but only move it (e.g., to underline textual information on the screen).
FOURTH GROUP: A fourth group of 2870 display modes works with the "Two Handed Dual Pointers" input mode, and provides display mode 2870 (a) with two visible objects: the native mouse cursor, e.g. as a white arrow, and a second arrow (e.g. black arrow), already described in FIG. 10 icon 2870 (b) represents a mode similar to mode 2850 (b), except that two bitmaps are shown with one hand, one movable with the left pointing device, the other movable with the right pointing device (e.g. as shown in Fig. 34); - icon 2870 (g) to icon 2870 (j) represents a mode similar to modes 2850 (g) to 2850 (j), except that the native mouse cursor will follow the position associated with the pointing device that was most recently moved , and / or whose button was most recently pressed, and / or whose scroll wheel was turned most recently, as explained above. These modes are particularly suitable for users who want to move and click and / or scroll both pointing devices, for example the transcribers of FIG. 4, but also, for example, the translators or proofreaders of FIG.1 who wish to scroll and / or edit through both documents, using both pointing devices.
As described above (see, e.g., FIG. 14 and FIG. 15 and FIG. 19 and FIG. 20), the modes of this fourth group 2870 work well even with classic pointing devices M1, M2 provided that the pointing device being clicked and / or scrolled is the one that was moved most recently, requiring special user attention. A fairly simple solution would be to always move a mouse manually before clicking it. But as explained above, these modes work even better when using special pointing devices (see, e.g., FIG. 22) that automatically take care of "move before clicking" and / or "move before scrolling", or with a special hub or adapter or dongle as described above, see eg FIG. 26 (a) to FIG. 26 (d).
FIG. 28 (b) shows an exemplary pop-up menu that can be accessed by clicking a right button of the dominant pointing device on some of the display modes in the mini-interface window of FIG. 28 (a). In the example shown, the menus allow to change the color, and size of the second and / or further visible object (s), as well as to select a transparency level of the click-through overlay window 584 (FIG.5c) or 884 (FIG 8c).
FIG. 29 shows another exemplary user interface window, referred to herein as the "main user interface," as may be used in embodiments of the present invention to configure the overlay application.
The user interface (UI) window shown in FIG. 29 has an area titled "Mouse / Touchpad" (English: "Mouse / Touchpad") for selecting the input mode (eg Left Handed or Right Handed or Two Handed), and for selecting which pointing device (or rather, which "handle) "assigned by the operating system to each pointing device) should be considered the dominant pointing device (or" handle "thereof). The U1 also includes components such as e.g. radio buttons for selecting relative speed of movement, etc. In FIG. 31 (a), this area is shown in an enlarged view.
The interface window shown in FIG. 29 also has an area entitled "Background"), which allows the click-through overlay window 584, 884 containing the at least one second visible object 506L, 806L to further contain a bitmap containing pixels 594, 894, a majority of which are fully transparent pixels, or semi-transparent monochrome pixels, or a semi-transparent texture bitmap as described in more detail in the co-pending "perforated bitmap application (s)".
The exemplary user interface window shown in FIG. 29 also has an area titled "Transparency", which allows to select or choose an alpha transparency level from the overlay window 584. For example, if the background pixels 584 and 884 are fully transparent pixels, the degree of transparency of the movable visible object can be changed in this way. The user can choose a compromise between showing an object that is very different from the background pixels on the one hand, and on the other hand, being transparent enough to read underlying textual information. The user interface window shown in FIG. 29 also has an area entitled "Mode" (Dutch: "Mode"), which is already described in FIG. 28. Depending on which display mode is selected, the main window may show an additional area such as the area "block" where the user can select, for example, a color, height, width and vertical offset. The interface window shown in FIG. 29 also has an area entitled "Examples", for using the display modes with default settings, but default settings can of course also be implemented differently. In FIG. 30, this area is shown in an enlarged view. It is noted that embodiments of the present invention are by no means limited to this particular user interface.
FIG. 30 is an enlarged view of a portion of FIG. 29. It provides a visible overview of several possible embodiments of the present invention, however, the invention is not limited thereto. These standard modes are largely similar to the modes shown in the mini-interface of FIG. 28 (a). Some of the modes are described in more detail in this document, as indicated by the Roman numbers referring to the respective figures. FIG. 31 (a) is an enlarged view of a portion of FIG. 29. It shows for example how a user can select an input mode, eg Left Handy; Right Handy; Two-handed. It is noted that, for example, a laptop computer touchpad is also considered a pointing device, hence the indication "# 3" in the title, which means that the operating system of the computer device from which this screenshot was taken has found three input devices: two physical mouse devices and one touchpad. It is noted that the present invention will also work, for example, with a laptop computer to which only a single physical mouse is connected, e.g., using the touchpad as the non-dominant pointing device and the physical mouse as the dominant pointing device. FIG. 31 (b) is a variation of the portion of FIG. 31 (a), adding an option to adjust the orientation of the displacement vector defined by the displacement values (dx, dy) to a preferred orientation selected from a first group of predetermined directions consisting of the directions: North, West, South, East, or from a second group of predetermined directions consisting of the directions: North, North-West, West, South-West, South, South-East, East, North-East. This transformation can be applied to movements of only the dominant pointing device, or to movements of only the non-dominant pointing device, or to both pointing devices.
Tests have shown that this function can greatly help the user to better control movements of one or both pointing devices when performing certain tasks. This function can be particularly useful, for example, to move the pointing device controlled by the non-dominant hand horizontally across a text line, e.g. when performing text-based tasks such as reading a text document, proofreading a translation, point to individual words on the same line during a Webinar or presentation, etc.
FIG. 31 (c) shows some lines of pseudo code that can be used in the position update routine to transform the displacement values dx, dy to be located in an orientation of the second group of eight allowed orientations mentioned above (North, North-West, etc.). In the example of FIG. 31 (c), - if dx = 0, it means that the movement was already vertical, therefore no adjustment is required; - if dy = 0, it means that the movement was already horizontal, therefore no adjustment is required; - if dx = dy, it means that the movement was already diagonal, therefore no adjustment is required, - otherwise, it is determined whether the angle of the displacement vector is in the range (-30 ° to + 30 °) or in the range of (150 ° to 210 °), in which case the absolute value of the tangent of said angle is less than 0.5, in which case the movement is forced to be horizontal, and / or it is determined whether the angle of the displacement vector is in the range of (60 ° to 120 °) or in the range of (240 ° to 300 °), in which case the absolute value of the tangent of said angle is greater than 2.0, in which case the movement is forced to be vertical, otherwise the movement is forced to be diagonal (oriented at + 45 ° or 135 ° or 225 ° or 315 °).
It is noted that no trigonometric functions are used in this routine, thus keeping the implementation fairly simple and fairly fast, and therefore keeping the CPU load quite low. The careful reader will recognize that the amplitude of the displacement vector may possibly increase or decrease slightly with the adjustment, in order to keep the transformation simple and fast. This effect is hardly noticeable.
Tests have shown that the transformation of FIG. 31 (c) slow motion of the pointing device (for which dx and dy are very small). To a user, this feels as if quite slow movements can be performed in any direction, which is desirable to allow fine-tuning of a position on the screen. However, if the speed at which the pointing device is moved is increased, the amplitude of the displacement vector will become greater than a predetermined threshold (greater than 0 in the example of FIG.31c), in which case the displacement is forced to be horizontal or vertical. or diagonal. This behavior is very desirable to keep near horizontal movements perfectly horizontal, which is difficult to achieve manually, especially at fairly high speed.
But of course, the present invention does not require this feature to be present (it can be enabled or disabled by a user). Furthermore, if present, the present invention is not limited to the specific routine or algorithm shown in FIG. 31 (c), and other algorithms for reorienting the pointer movements have also been considered, as described, for example, in the following variants.
FIG. 31 (d) and FIG. 31 (e) show a variation of the displacement transformation routine, in which near-horizontal motions are transformed into purely horizontal motions.
More specifically, FIG. 31 (d) A graphical representation of the displacement vector V corresponding to the physical motion of the pointing device associated with the displacement values (dx, dy). The displacement vector V forms an angle α with respect to the X axis. If this angle lies in the segments indicated in gray, which means that the angle a is in the range [-amax.…. + Amax] or in the range [180 ° -amax .. 180 ° + amax], then the displacement vector is transformed into a purely horizontal displacement, by setting the value of dy equal to zero. Rather than working with angles, this can be codified by calculating the absolute value of the tangent of the angle, that is abs (dy / dx), and testing whether this value is less than or less than or equal is at a predetermined constant. For example, amax = 30 ° corresponds to abs (dy / dx) = 0.577, or as another example, testing whether abs (dy / dx) <0.5 corresponds to testing whether the angle α is in the range from about [-26 ° to + 26 °] or about [154 ° to 206 °], but of course the present invention is not limited to this range only, and other ranges may be used as well.
FIG. 31 (e) shows exemplary lines of pseudo code implementing the "displacement transform" described above, which can be used in the "position update routine" (see, for example, step g) of FIG. 5 (e), when updating Pos1 and Pos2) But of course a different angular range can be used, or in other words the invention will also work with other values for MAXTAN (in FIG. 31e), for example values in the range of 0.2 to 0.75, for example smaller than 0.25 = 1/4, or less than 0.33 = 1/3, or less than 0.5 = 1/2, or less than
0.66 = 2/3, or less than 0.75 = 3/4. One skilled in the art having the benefit of the present invention can easily define other suitable ranges. FIG. 32 shows a computer system 3200 according to an embodiment of the present invention, comprising a computer device 3201 according to an embodiment of the present invention executing an overlay application according to an embodiment of the present invention, and two pointing devices, e.g., mouse devices 3203L, 3203R connected to the computer device, and optionally a keyboard 3202. The mouse devices may be classic mouse devices, or may be special mouse devices with the "move-before-click" function and / or the "move-before-scroll" function as explained in FIG. 22 (a) and FIG. 22 (b). The mouse devices may be connected directly to the computing device (e.g., as shown in FIG. 11 and FIG. 23 (a)), or via a classic hub or adapter or dongle (e.g., as shown in FIG. 12 and FIG. 24) or through a dedicated hub (e.g., as shown in FIG. 25).
The overlay application used in the system of FIG. 32 allows the O / S and GUI (eg.
Windows 7 or Windows 10 from Microsoft Corporation, or mac OS Sierra or mac OS High Sierra from Apple Inc.) displays the native mouse cursor 3205, but adjusts its position, e.g. overwrites it. The overlay application further provides a second visible object 3206L and a third visible object 3206R, in the example of FIG. 32 being two lines or rectangular areas. These rectangular areas can be opaque or semi-transparent. They can have a single color, or a color gradient, or can contain a perforated bitmap. The other pixels of the overlay window may be fully transparent pixels, or may be all semi-transparent pixels, or the overlay window may include a perforated bitmap or a texture bitmap as explained in the co-pending "perforated bitmap application (s)".
In the event that the overlay application is designed (e.g. hard-coded) for, or configured in display mode 2860 (h), i.e. in the Right Handy input mode, the object 3206L will follow movements of the left pointing device 3203L (or rather the pointing device configured or referred to as the left pointing device) and the native mouse cursor 3205 and object 3206R track movements of the right pointing device 3203R (or rather the pointing device configured or referred to as the right pointing device).
In the event that the overlay application is designed (e.g. hard-coded) for, or configured in display mode 2870 (h), i.e. in Two-Handed input mode, then the object 3206L follows movements of the left pointing device 3203L (or rather the pointing device configured or referred to as the left pointing device) and the object 3206R tracks movements of the right pointing device 3203R (or rather the pointing device configured or referred to as the right pointing device), and will move and / or jump the native mouse cursor 3205 depending on which pointing device most recently was moved and / or clicked and / or scrolled.
The embodiment shown in FIG. 32, and variants thereof, optionally using a wired or wireless hub or adapter or dongle, and optionally using a motion transform as explained in FIG. 31 (b) and FIG. 31 (c), may be particularly helpful in addressing the problem described in FIG. 1 and FIG. 3, where it is desirable to underline two text fragments.
FIG. 33 shows embodiments of the present invention implementing display mode 2850 (g), 2860 (g), and 2870 (g), the O / S and GUI showing the native mouse cursor 3305, the shape of which typically can and will change depending on the context, and the overlay application additionally shows two visible and movable squares 3306L, 3306R. Display mode 2860 (g) and 2870 (g) appear to be particularly suitable for each of the cases shown in FIG. 1 to FIG. 4 for right-handed people, while display mode 2850 (g) and 2870 (g) appear to be particularly suitable for similar problems as shown in FIG. 1 to FIG. 4, but for left-handed people.
FIG. 34 shows embodiments of the present invention implementing display mode 2870 (b), where the O / S and GUI show the native mouse cursor 3405 (in the example a white arrow}, and the overlay application additionally shows two visible objects, a second object 3406L being or containing a left hand bitmap or image, and a third object 3406R being or containing a right hand bitmap or image Also this mode appears to be particularly suitable for all of those shown in FIGS. 1 to FIG. cases, especially in combination with a motion transformation routine as explained in FIG. 31 (b) and FIG. 31 (c) to help the user to make horizontal movements easily, and especially if the hands are semi-transparent. in this case, the position of the bitmap containing the image of the hand is slightly offset from the position of the native mouse cursor 3405 such that the native mouse cursor is on or near the end of the respective index fingers. Tests have shown that the two hands can be extremely helpful in proofreading.
Although the present invention can also be used with bitmaps other than a hand, it is noted that the shape of a hand, particularly a hand with an outstretched index finger, feels very natural and intuitive, and does not diminish the user's attention. perhaps because it is much like the real one, in FIG. 1 to FIG. 3 hand and finger shown. That being said, a triangular shape, or a polygonal shape, preferably with a pointed end at the top, can also be used.
FIG. 35 shows embodiments of the present invention implementing display modes 2850 (d) and 2860 (d), showing the native mouse cursor 3505 as a first visible and movable object, and one line 3506 or rectangle as the second visible and movable object. These modes can be particularly helpful in addressing the problem shown in FIG. 2 and FIG. 3. It may be an advantage of this embodiment that there is no third object following the native mouse cursor that could be slightly distracting or slightly annoying when selecting menus or colors or the like.
FIG. 36 shows embodiments of the present invention implementing display modes 2850 (e) and 2860 (e), showing the native mouse cursor 3605 as a first visible and movable object and one small cross 3606 as a second visible and movable object. These modes can also be particularly helpful in addressing the problem shown in FIG. 2 and FIG. 3. Display modes 2850 (j) and 2860 (j) showing a large cross extending the entire height and width of the overlay window can be viewed as a variation of the embodiment shown in FIG. 36. These modes can be particularly useful when reading or extracting information from a worksheet with the non-dominant hand while working with the native mouse cursor in another document, such as a text document. The large cross overlay offers the advantage that a specific cell can be easily found, and that the row and column headers of a particular cell are also extremely easy to find, as explained in more detail in the co-pending "perforated bitmap application ( n) ". For the sake of completeness, it is noted that the single mouse mode 2840 (f) may be better suited when editing a worksheet, because in this embodiment the native mouse cursor is located at the intersection of the large cross. That being said, with the benefit of the present disclosure, those skilled in the art can easily add a display mode where the display mode cross 2840 (f) is used as the third visible object, and another object, for example, a line or a square or a hand becomes. added as a second visible object. FIG. 37 is a variation of FIG. 32 which implements the same modes, but is used to underline different parts of a single document (virtually). FIG. 38 illustrates embodiments of the present invention implementing display modes 2850 (i) to 2870 (i) showing the native mouse cursor and two small crosses 3806L, 3806R.
FIG. 39 shows embodiments of the present invention where the overlay application is configured in one of display modes 2850 (j), 2860 (j), and 2870 (j). The horizontal line or bar 3906L located on the left side of the vertical line 3907 is automatically moved up / down when moving the left pointing device 3903L. The horizontal line or bar 3906R located on the right side of the vertical line 3907 is automatically moved up / down when moving the right pointing device 3903R. Depending on which input mode is selected or implemented, the native mouse cursor 3905 tracks movements of the left pointing device 3903L (in the Left Handed input mode), or of the right pointing device 3903R (in the Right Handing input mode), or of the pointing device that was most recently moved and / or clicked and / or scrolled (in Two-Handed input mode). An advantage of this embodiment for proofreaders is that the non-dominant hand does not require precise positioning, since an entire line of text can be underlined by just moving the non-dominant pointing device up / down. Left and right movements of the non-dominant pointing device are not required. This is especially true for "simple texts" that can be read fairly quickly, where it is not required or desirable to underline each word individually, unlike, for example, translations of patent claims, where each word must be carefully checked.
FIG. 40 shows a variation of the one shown in FIG. 39 embodiments shown, wherein the overlay application, in addition to the horizontal line or bar on the left and right sides of the vertical line 4007, also shows two blocks or squares or rectangles 4009L, 4009R within the horizontal line or bar 4006L, 4006R, which blocks along with the lines move in the vertical direction (up / down), but are also movable in the horizontal direction (left / right) in accordance with movements of the respective indicating devices 4003L, 4003R. Typically, the line or bar will have a different color from the blocks or squares or rectangles within the line or bar so that they are clearly visible. In a specific embodiment or configuration, the lines have a monochrome color and the blocks are completely transparent, giving a visual impression of a beam with a "hole".
The inventors also got the idea to solve the problem shown in FIG. 1 to FIG. 3 in a completely different way, namely, outside the computer device, for example in a special display device 4104, as shown in FIG. 41.
FIG. 41 shows a computer system 4100 according to an embodiment of the present invention, including a dedicated monitor or display screen 4104 according to an embodiment of the present invention, and a computer device 4101, which may be a conventional computer device. The computer system 4100 includes at least two pointing devices, including a first pointing device 4103R connected to the classical computing device 4101, and a second pointing device 4103L connected to the dedicated display screen 4104, in a wired or wireless manner, e.g. via a VGA cable or a DVI cable or an HDMI cable 4199 over which uncompressed video is sent.
In simple terms, the display 4104 has a processor adapted to communicate with the pointing device 4103L and to maintain a position xL, yL corresponding to movements of the pointing device 4103L, and adapted to overlay the graphic image. received from the computer device 4101 with a locally generated visible object 4106L, which is, for example, in the shape of a square or rectangle or rectangle with rounded edges, or a circle or an arrow or a hand or a small cross or a large cross located extends the entire height and width of the display screen, or other suitable shape, and for moving that object in accordance with movements of the pointing device 4103L by positioning the object 4106L at the maintained position xL, yL. In other words, in this embodiment, the display 4104 generates the object 4106L and displays it on the screen at a position movable by the left pointing device 4103L, while the computer device handles movements of the right pointing device 4103R as usual.
FIG. 42 shows an exemplary block diagram of an embodiment of the display device 4104 as may be used in the computer system 4100 of FIG. 41. The display device 4104 includes: an input port 4231, e.g. a VGA port or an HDMI port or a DVI port or other suitable port for receiving an uncompressed video signal including graphic image data from a computer device 4101 (so no MPEG2 or MPEG4 compressed video stream); an interface or port for communicating with the pointing device 4203L, e.g., a physical connector 4237 such as a USB connector, or a wireless RF interface 4238; a display panel 4236, e.g., an LCD panel for generating a visible image; an input buffer 4232 for storing at least a portion of the graphic image data received at the input port 4231; a frame buffer 4234 for storing at least a portion of the image data to be displayed on the display panel 4236; and a processor adapted to: * communicate with the pointing device 4203L to receive displacement values dx, dy related to movements of the pointing device; * maintaining a position xL, yL based on the received displacement values dx, dy; * generating image data of the visible object 4106L; generating the image data to be stored in the frame buffer 4234 by overlaying the image graphic data received at its input port with the generated image data of the visible object 4106L at the maintained position xL, yL.
In the case of a VGA or other analog input, the incoming signal must be digitized by means of one or more ADC (analog-to-digital) converters, in ways known per se in the art. Likewise, if the output signal is analog, then the digital signal leaving the output buffer must be converted to analog signals using one or more DAC (digital-to-analog) converters, in ways well known in the art. . The display device may optionally include input signal conversion circuitry for converting analog signals to digital signals, and / or may optionally include output signal conversion circuitry for converting digital signals to analog output signals.
The pointing device 4203L can be a wired mouse device or a wireless mouse device. The pointing device 4203L may be an HID compatible pointing device, e.g., an HID compatible mouse device. The pointing device can be connected directly to the physical connector via wires, or via a classic hub (not shown).
The pointing device may be a wireless pointing device, e.g., a wireless mouse device, that communicates via a dongle connected to, e.g., inserted into the physical connector of the display screen.
The display screen may include an internal or external RF wireless transceiver 4238 for communicating with a wireless pointing device, e.g., via a Bluetooth protocol.
The generated visible object 4106L may be in the shape of a square, or a square with rounded edges, or a rectangle, or a rectangle with rounded edges, or an elongated object, or an arrow or a hand, or other suitable shape.
The object 4106L may be monochrome pixels, e.g., red or blue or black or yellow pixels, or some other suitable color. Alternatively, the object may have exactly two colors, e.g., an inner, inner color, and an outline color (e.g., black), e.g., to create greater contrast. But the visible objects can also contain more than two colors.
The object 4106L may be opaque, that is, the user cannot see through the object 4106L. In this case, the processor can generate the overlaid image by simply replacing some of the graphic image data from the computer with image data from the locally generated visible object, without having to mix data.
The article 4106L may be a semi-transparent monochrome article, e.g. having an alpha transparency α in the range of 10% to 90%, e.g. about 25% or about 50% or about 75%. In this case, the processor will alpha-blend or mix the graphic image data with the object data. In the case of 50% alpha blending, the red, blue and green color components R1, G1, B1 of the image graphic data and the red, blue and green color components R2, G2, B2 of the pixels of the movable object can be easily averaged out, resulting in a simple arithmetic operation is, for example, equivalent to the following calculation: R: = (R1 shr 1) + (R2 shr 1), and likewise for the other color components.
The article 4106L may be or comprise a so-called "perforated bitmap" as explained in the co-pending "perforated bitmap application (s)". This bitmap may include a first plurality of fully transparent pixels and a second plurality of monochrome pixels, wherein the first plurality of pixels and the second plurality of pixels are interleaved (alternately positioned), e.g., in a chessboard pattern. * This perforated bitmap can be overlaid in an opaque manner, as follows: in the event that a graphic image pixel from the computer image is overlaid with a fully transparent pixel, the result of the overlay will be the original graphic image pixel.
In the event that a graphic image pixel is overlaid with a monochrome pixel, the result of the overlay will be the monochrome pixel of the movable object. * This perforated bitmap can be overlaid in a semi-transparent manner, as follows: In case a graphic image pixel is overlaid by a fully transparent pixel, the result of the overlay will be the original graphic image pixel.
In the event that a graphic image pixel is overlaid with a monochrome pixel, the result of the overlay will be the alpha blending or blending of the graphic image pixel from the computer image and the monochrome pixel of the movable object.
It is also possible to make the object 4106L appear semi-transparent by time multiplexing between the graphic image pixels (coming from the computer device), and the image pixels of the movable object (generated within the display screen). For example, using / displaying the monochrome pixels of the object 4106L at a first time in time (e.g., at even-numbered frames) and using / displaying the graphic image data at a second time in time (e.g., at odd-numbered frames) frames). For example, this would mean that half of the frames would effectively show the visible object 4106L, and half of the frames would not, creating the impression of being semi-transparent (averaged over time). Comparison of FIG. 41 with FIG. 5 (a) and FIG. 5 (b) teaches that the images on the screen can look exactly the same or very similar (assuming the same color and / or the same semi-transparency level are used). Thus, the embodiment of FIG. 41 also the view shown in FIG. 1 to FIG. 3, but the computing device 4101 does not require the special overlay application described above and / or the dedicated mouse and / or the dedicated hub, only the display 4104 must be "special".
The computer system 4100 of FIG. 41 allows the user to move the left pointing device 4103L to move the position of the visible object 4106 on the screen, but of course clicking that pointing device or scrolling its wheel will not have the effect of selecting text or scrolling up or down a document. In other words, the embodiment of FIG. 41 does not solve the problem shown in FIG. 4.
In a variation of the display device shown in FIG. 42, the display comprises at least two connectors for connecting at least two pointing devices, and the processor 4233 is adapted to maintain two pointing positions Pos1, Pos2, and to generate two visible objects, for example two lines similar to FIG. 32, or two squares similar to FIG. 33 or two hands similar to FIG. 34, and for positioning these two objects at the two tracked locations. For example, with a computer system including such a display device, a user could use two dedicated pointing devices for comparing two text documents (e.g. when proofreading a translation}, and use a third pointing device connected to the computer 4101, for editing or correcting.
The display device 4104 may have a further connector for connection to a removable, non-volatile memory device, such as a memory card or a USB stick or the like, which may contain a file and a bitmap of the one or more visible objects to be displayed. .
The inventors came up with yet another idea to solve the problem shown in FIG. 1 to FIG. 3, also outside the computing device 4301, namely by providing a special video adapter or video interface device 4330 connected between the computing device 4301 and the display device 4304, as shown in FIG. 43 and FIG. 44.
Many types of video adapters are known in the art, for example a video adapter for converting an HDMI signal to a VGA signal or vice versa. The video adapter may contain one or more high-speed analog-to-digital converters ADC and / or one or more high-speed digital-to-analog converters DAC.
None of the existing video adapters maintains a position related to movements of a pointing device associated with said adapter, and none of them shows a visible object movable in accordance with these movements. Preferably, the video adapter 4330 according to embodiments of the present invention includes a digital video input connector (e.g., an HDM1 input connector) configured to receive an uncompressed video signal (e.g., no MPEG2 or MPEG4 stream), and a digital video. - output connector (e.g. an HDMl output connector), in which case the incoming video or graphic image data is digital, and also the outgoing video or graphic image data is digital, and the DAC (s) and the ADC (s) can be omitted turn into. But a video adapter with a VGA input connector is also possible.
FIG. 43 shows a computer system 4300 according to an embodiment of the present invention, including a video adapter device 4330 according to an embodiment of the present invention.
In the context of the present invention, the "video adapter" is an electrical or electronic device in a single housing (e.g., box) with a typical length of 4 to 14 cm, and a typical width of 4 to 12 cm, and a typical height of 1 to 4 cm, and therefore not a laptop computer or desktop computer, and certainly not a computer system with several network servers. The device 4330 preferably does not have a separate power cord or power supply, but is preferably powered from the video input port A directly from the computer 4301.
Thus, the video adapter 4330 has a video input port A, 4431 (e.g., a DVI or an HDMI port or a VGA port) for receiving an uncompressed video signal with first graphic image data from a computer device 4301. The computer device 4301 is connected to a first pointing device 4303R. The video adapter 4330 further includes a video output port C, 4436 (e.g., a DVI or an HDMI port or a VGA port) for providing an uncompressed video signal with second graphic image data to a monitor or display screen 4304. The video adapter 4330 further includes at least one interface (e.g., a physical connector such as a USB port, or an RF transceiver) for communicating with a second pointing device 4303L operably connected to the video adapter. The video adapter 4330 is adapted to generate at least one visible object 4306L, for example in the shape of a square or a rectangle or a rectangle with rounded edges, or a circle or an arrow or a hand or a small cross or a large cross that extends the entire height and width of the display screen, or any other suitable shape, and for generating the second graphic image (to be sent to the display screen) by overlay the first graphic image data (upcoming of the computer device) with the at least one visible object 4306L (internally generated), and for moving that at least one visible object 4306L in accordance with movements of the second pointing device 4303L.
The at least one visible object 4306L can have a monochrome color. The monochrome color can be predetermined, for example red or green or blue or black or some other suitable color. Optionally, the video adapter 4330 may have one or more buttons or switches 4441 for selecting the monochrome color from a number of predetermined colors.
The video adapter is adapted to generate at least one visible object 4306L and to overlay the at least one visible object with the incoming image image data, e.g., by alpha blending or otherwise mixing or combining the first image image data from the image data. computer with the image data of the movable object. The at least one visible object can be an opaque or a semi-transparent object, or it can have a so-called perforated bitmap, as described in more detail in the co-pending "perforated bitmap application (s)". It can be understood by comparing FIG. 41 and FIG. 43 which also includes the embodiment of FIG. 43 the problems described in FIG. 1 to FIG. 3 solves.
FIG. 44 shows an exemplary hardware block diagram of a video adapter 4330 as may be used in the computer system 4300 of FIG. 43, but the invention is not limited thereto. The video adapter 4330 of FIG. 44 comprises the following components: a video input port 4431, e.g. in the form of a VGA connector or DVI connector or HDMI connector, for receiving an uncompressed video signal with first graphic image data from a computer device; a video output port 4436, e.g. in the form of a VGA connector or DVII connector or HDMI connector for providing an uncompressed video signal with second graphic image data to a display device; an input buffer 4432 for storing at least a portion of the first graphic image data received at the input port; a processor 4433 connected to the input buffer 4432, and adapted to generate second graphic image data based on the first graphic image data; a wired and / or a wireless interface 4437, 4438 for communicating with a wired or a wireless pointing device 4403L; wherein the processor 4433 is further adapted to: * communicate with the pointing device 4403L to receive displacement information dx, dy related to movements of the pointing device 44031; * updating a position Pos2 based on the received displacement information dx, dy; * generating a visible object 4306L; generating the second graphic image data by overlay the first graphic image data with the generated visible object 4306L at the position Pos2.
The video adapter may further include a frame buffer 4434 operably connected to the processor 4433, adapted to store at least a portion of the first graphics image data received from the input buffer 4432, and to store at least a portion of the second graphics data. to be sent to output buffer 4435.
However, a frame buffer is not absolutely required, e.g. in case the incoming video stream and the outgoing video stream have the same frame rate and resolution. In such embodiments, the video adapter's volatile memory (RAM) is preferably less than one frame memory.
Optionally, the video adapter 4330 may further include one or more buttons 4441, for example, to enable the user to select a color and / or shape and / or size of the movable object 4306L.
Optionally, the video adapter 4330 may further include one or more indicators, for example in the form of LEDs 4442, to provide information to the user.
The video adapter 4330 may include a memory device (not shown) that includes the input buffer and the output buffer and (if present) the frame buffer.
However, the present invention is not limited to this particular embodiment, as other architectures can fulfill the same functionality. For example, if the clock speed of the incoming video data and the outgoing video data is the same, then the second graphic image can be generated in real time, and then the frame buffer can be omitted, and the generation of the visible object 4306L can be practically implemented in real time, along with the overlay or alpha merge operation. One skilled in the art of video chip design knows how to implement this functionality, but as far as is known there is no reason known in the art to provide pointing device support on such chips.
The video interface device 4330 may have additional interfaces, for example an additional USB port 4439 for connection to, for example, a USB memory stick that may include, for example, a bitmap of the movable object, and / or may further include a texture bitmap to be overlaid with the incoming video data, for example in a manner as described in the co-pending "Perforated Bitmap Application (s)".
In one embodiment, the video interface device 4330 has an integrated RF transceiver 4438 for communicating with a wireless mouse and / or for communicating with the personal computing device, for example, to configure the video adapter device 4330, and / or for selecting the size or color of the object, etc. But such a transceiver can also be provided externally, e.g. as a dongle connected to the connector 4437 or 4439.
In a variation of the video adapter device shown in FIG. 43 and FIG. 44, the video adapter includes at least two connectors for connecting to at least two pointing devices, and the processor 4433 is adapted to maintain two pointing positions Pos1, Pos2, and to generate two visible objects, for example two lines as shown in FIG. 32, or two squares as in FIG. 33 or two hands as in FIG. 34, and for positioning these two objects at the two tracked locations. For example, with a computer system including such a video adapter, a user could use two dedicated pointing devices to compare two text documents (e.g. when proofreading a translation}, and use a third pointing device connected to the computer 4301 , for editing or correcting.
The video adapter device 4330 may optionally be further provided to convert fairly bright image information to fairly dark image information and vice versa. In doing so, incoming pixel values (Rin, Gin, Bin} are converted to outgoing pixel values (Ruit, Guit, Buit). This can be done as follows: First, (Rin, Gin, Bin) is converted to YUV values, e.g. by means of of the formulas: Y = 0.299R + 0.587G + 0.114B; U = 0.492 (BY); V = 0.877 (RY), or by means of the formulas: Y: = (R + G + G + B) shr 2 ; U: = (BY) div 2; V: = (RY), or by means of formulas with slightly deviating coefficients, after which the outgoing RGB values are calculated using one of the following functions: a) YUV2RGB (255-Y , U, V, Square, Guit, Buit), or b) YUV2RGB (255-Y, U div 2, V div 2, Square, Guit, Buit), or c) YUV2RGB (255-Y, -U, -V , Square, Guit, Buit), or d) YUV2RGB (255-Y, -U div 2, -V div 2, Square, Guit, Buit), where the conversion of YUV values to RGB values can be calculated, for example by the following formulas: R = Y + 1.140V; G = Y - 0.395U - 0.581V; B = Y + 2.032U, or by the formulas: R: = Y + V; G: = Y - ((U + V) div 2); B: = Y + (U + U), or by means of formulas with slightly different coefficients.
Formula (a) and (b) have the advantage of virtually retaining the original color, which is useful for displaying documents with a particular color convention (e.g. black for conclusions, blue for benefits, etc.).
Formula (c) and (d) have the advantage that not only the intensity is inverted, but also the colors, which is more readable for some applications (such as some email programs).
The formulas with floating-point operations naturally require more processing power. The formulas with integer operations can usually be executed faster, or on a less powerful processor. Tests have shown that the results of these four solutions are aesthetically pleasing and can potentially reduce eye fatigue, especially when the original images have a typically white background.
Tests have shown that the following set of formulas also give very good results: a) Y: = (R + G + G + B) div 4; b) Rout: = (256-Y) + A. (R-Y), bounded to the region [0.255]; c) Gout: = (256-Y) + B. (G-Y), limited to the range [0.255];
d) Bout: = (256-Y) + C. (B-Y), limited to the range [0..255]; where (R, G, B) is the red, green, and blue value of incoming pixels, and (Rout, Gout, Bout) is the red, green, and blue value of outgoing pixels, and where A, B, C are preceded certain constants.
Preferably the constants A, B, C are equal to each other.
Preferably, these constants are rational numbers in the range -7/8 to +7/8.
In one embodiment, A, B and C are values selected from the following set: -1/2, +1/2, -1/3, +1/3, -2/3, +2/3, -1/4 , +1/4, -3/4, +3/4, -1/5, +1/5, -2/5, +2/5, -3/5, +3/5, -4/5 , +4/5, -1/6, +1/6, -5/6, +5/6, -7/8, +7/8.
Several months after the solutions described above, and searching if there would be no software solution to the problem of FIG. 21 without special hardware (i.e. without a dedicated mouse or a dedicated hub), but without giving up the convenient native mouse cursor skipping, the inventors came up with the idea that the functionality of the “dedicated hub” described above can also be implemented in a so-called "mouse filter driver" (sometimes abbreviated herein as MFD). Although this embodiment will not be further described, the present invention is also directed to this embodiment. is thus based on the principle of pre-positioning the native mouse cursor correctly, before sending a click or scroll message, by using a dummy motion message, and by delaying (delaying) the original message.
The inventors also came up with another idea which will now be described with reference to FIG. 45 to FIG. 54.
FIG. 45 shows a computer system 4500 with a computer device 4501 that can be viewed as a variant of the computer system 1100 of FIG. 11, to which is added a special mouse filter driver 4563, which is provided to: i) detect motion messages from the first and second pointing devices 4503R, 4503L, and upon detection of such message, send the motion message or a corresponding message to control the 4560 control system; and ii) at least one message from the group consisting of a button-pressed message, a button-released message, a scroll-up message and a scroll-down message from the first and / or second pointing device 4503R. Detect 4503L, and upon detection of such message, send a first, second, third or fourth (predetermined, or parameterizable or at least partially predetermined) motion sequence to the control system (O / S), respectively; and wherein the special overlay application 4551 is further provided to:
- detect these motion sequences (or corresponding motion message sequences), - and upon detection, first (re) position the native mouse cursor 705 to the position corresponding to the tracked position for the mouse device that caused the sequence in question, - and shortly after, preferably immediately after, or even better in the same command, a button-pressed command, or a button-released command, or a scroll-up command or a scroll-down command to be sent from the overlay application 4551, from the correct position, to the operating system, which in turn will pass the command to the appropriate underlying application (because the overlay application is configured in click-through mode, and / or is essentially completely transparent), more specifically to the application (or visual component) that is "below" the left mouse position (Pos2) when the left mouse was operated (scrolled or clicked) or the application (or vi suele component) that is located "below" the right mouse position (Pos1), when the right mouse button has been operated (scrolled or clicked).
It is noted that the mouse filter driver 4563 does not need to know which is the first and second pointing device. In an embodiment where more than two pointing devices are connected to the computing device 4501, the mouse filter driver sets all "scroll up", "scroll down", "button pressed" and "button released" messages. from all mouse devices, to a corresponding (e.g., predetermined or parameterizable) motion sequence. The overlay application can decide which of these sequences come from the first or second pointing device, e.g. as the right mouse and the left mouse.
In terms of terminology, a "scroll up message" and a "scroll down message" are sometimes simply referred to herein as "scroll message", and are referred to as a "scroll up command" and a "scroll down message". down command "sometimes simply referred to as a" scroll command ", without specifying which of the two (up or down). By the way, by "scroll up message" is meant a scroll message with one or more parameter (s) that should cause an upward scroll. The skilled person knows what is meant by this.
Similarly, in the explanation below, the phrase "button pressed" and "button released" message are used, without reference to which button it is.
Although the invention is primarily explained for a mouse with one scroll wheel that can be turned (scrolled) up or down, and with one push button that can be pressed or released, and with four corresponding movement sequences, this principle can of course be extended to a pointing device. with multiple scroll wheels (e.g. a horizontal scroll wheel) and / or with multiple push buttons, e.g. a mouse with exactly one scroll wheel and exactly two push buttons (sometimes referred to herein as left button and right button).
In a preferred embodiment, the mouse filter driver 4563 is provided to recognize, in addition to regular mouse movements (moves), at least six or exactly six commands: (1) scroll up, (2) scroll down, (3) Left button-pressed, (4) Left button released, (5) Right button pressed, (6) Right button released, and to send one of six corresponding, e.g. predetermined or parameterizable or partially predetermined motion sequences. The "left button" is sometimes also referred to as "button1", and the right button as "button2", but the invention can be easily extended to implementations with more than two buttons and more than one scroll wheel.
MOUSE FILTER DRIVER: Simply stated, the mouse filter driver 4563 should perform the following functions, or a subset of them.
a) detect a motion message (also called a move message) originating from a mouse device, and when detected, send the motion message (or a corresponding message or command) to the O / S, e.g. to an input buffer of the O / S S, possibly in a different format and / or at a slightly different time (eg at the request of the O / S); b) detecting a button-pressed message from a mouse device, and when detected, sending a first predetermined sequence of motion messages to the input buffer of the O / S in place of the button-pressed message; c) detecting a button released message from a mouse device, and when detected, sending a second predetermined sequence of motion messages to the input buffer of the O / S in place of the button released message; d) detect a scroll-up message from a mouse device, and when detected, send a third predetermined or parameterizable or partially predetermined sequence of motion messages to the input buffer of the O / S, replacing the scroll -up message; e) detect a scroll-down message from a mouse device, and when detected, send a fourth predetermined or parameterizable or partially predetermined sequence of motion messages to the input buffer of the O / S, instead of the scroll - down message.
It is noted here that the terms "third" and "fourth" etc. do not have an absolute meaning, but only serve to distinguish between the different sequences that are or may be used. For example, a different sequence is preferably used for pressing the left button (also called button 1) and pressing the right button (also called button 2), and for releasing the left button and releasing the right button, but not for the sake of description. To make it unnecessarily complicated, this is usually made abstract.
However, if the pointing device has multiple buttons (a typical mouse has at least a left button and a right button), then further (e.g. predetermined or parameterizable) motion sequences may be used, e.g. a seventh sequence, an eighth sequence, etc.
If the pointing device has multiple scroll wheels (e.g. a horizontal scroll wheel and a vertical scroll wheel), then further (e.g. predetermined or parameterizable) motion sequences can be used, e.g. a ninth sequence (for scroll up of scroll wheel2), a tenth sequence (for scroll down of scroll wheel2), etc.
What is important is that the mouse filter driver converts a particular message into a particular sequence of (false) movements, and that the overlay application can translate this back to the appropriate button or scroll wheel operation. As far as is known, such translation is not used in the prior art. FIG. 51 shows an example of a method 5100 that can be performed by such mouse filter driver 4563. At step 5101, a message is received from a pointing device, (e.g., a message from a touchpad or a mouse, e.g., through a hardware interface and a hardware abstraction layer and a mouse driver 4561); In step 5102 it is checked whether this message is a motion message, and if so, in step 5103 the motion message (or a corresponding data structure) is sent to the operating system O / S, otherwise it is checked in step 5104 whether the received message is a button-pressed message was, and if so, a first (e.g. predetermined) motion sequence is sent to the operating system in step 5105, otherwise it is checked in step 5106 whether the received message was a button released message, and if so, then step 5107 sends a second (e.g. predetermined) motion sequence to the control system, otherwise, step 5108 checks whether the received message was a scroll-up message, and if so, step 5109 sets a third (e.g. predetermined) motion sequence is sent to the control system, otherwise it is checked in step 5110 whether the received message was a scroll-down message, and if so, a fourth (e.g. predetermined) message is sent in step 5111. weighting sequence sent to the control system. Steps 5108 to 5110 are optional. These are not needed, for example, if the computer device had two touch pads, or a touch pad and a stylus, but no scrolling pointing device.
As described above, step 5104 may include checking a Left Button Pressed message, and step 5106 may include checking a Left Button Released message, and the method may have additional steps (not shown in FIG. 51) for a Right Button Pressed message, and a Right Button Released message.
The order of the steps can of course be changed.
Preferably, the mouse filter driver 4563 imparts the same device handle to the message or data structure sent to the operating system O / S. This allows the overlay application 4551 to determine which pointing device triggered the sending of the particular motion message or sequence of motion messages (i.e., the motion sequence).
Experiments have shown that certain control systems do not transmit the movement sequence itself, but double the individual movement values, at least for some mice. The reason for this is not 100% clear, but this problem (or behavior) can be easily overcome by testing in the overlay application 4551 whether the "normal" motion sequence occurs, or the "double motion sequence" (ie, a sequence where each dx and dy value is double the original predetermined sequence).
It has also been found that certain operating systems seem to ignore the handle or pointer provided by the mouse filter driver, and merely look at the message content when it queries the various mouse drivers (English: polling). The mouse filter driver 4563 can also easily overcome this problem (or behavior) by passing the individual move messages only when the particular mouse driver 4561 that triggered the sequence is interrogated. A person skilled in the art with experience in the field of device drivers, and in particular with mouse filter drivers, and who takes advantage of the present disclosure, can implement such a device driver in a relatively simple manner.
For example, in a simple implementation, each "scroll-up" message will result in the third predetermined set of motion values, regardless of the value that the scroll wheel was rotated. For example, in another implementation, the mouse filter driver 4563 can send the same sequence several times in succession to the operating system. In yet another implementation, multiple different motion sequences can be used corresponding to different scroll wheel values. These different series can, for example, all start with the same sub-sequence, for example to facilitate recognition, etc.
It is a major advantage that the proposed solution of converting button messages and scroll messages to specific motion sequences works with existing mouse device drivers, and that these mouse device drivers 4561 themselves do not need to be modified. (Unless otherwise noted, "mouse driver" means "mouse device driver").
Preferably, the mouse filter driver 4563 is provided to run in kernel mode.
In certain embodiments of the present invention, the mouse filter driver 4563 has an active mode, in which it converts certain messages to motion sequences as discussed above, and a passive mode, in which it converts all messages (motion messages as well as scroll messages and button-presses). messages and button released messages) unaltered to the operating system. Such mouse filter driver is provided to cooperate with an overlay application that is provided to selectively activate (e.g. activate in dual mouse mode) and deactivate (e.g. in single mouse mode) the mouse filter driver. mouse modes, or when the overlay application is closed). In other embodiments of the present invention, the mouse filter driver is always active (see below).
OVERLAY APPLICATION: Simply stated, the dedicated overlay application 4551, e.g. its message processor 4552, must analyze the incoming message stream to determine whether any of the (supported, e.g. predetermined or parameterizable) motion sequences occur, e.g., in the total message stream, or in partial message streams associated with the individual pointing devices. This can be implemented e.g. by one or more state machines (e.g. one for each individual pointing device, or one for the first pointing device and one for the second pointing device), or by e.g. keeping track of the last N motion messages in one or more multiple buffers (e.g. one for the first pointing device and one for the second pointing device), or by e.g. converting the last N motion messages to characters, and keeping them in one or more text strings (e.g. one for the first pointing device and one for the second pointing device), or in other suitable ways. FIG. 48 shows an exemplary implementation, in pseudo-code, of a highly simplified example, how the overlay application, after the detection of a scroll-up sequence, can (re) position the mouse cursor, and can itself a scroll-up command. send to the operating system O / S. As an example, (re) positioning the native mouse cursor in a Microsoft Windows10 environment can e.g. be implemented using the setCursorPos () function, and emulating a scrolling gesture or a button press or a button release. e.g. implemented using the function MOUSE_EVENT (), with the first parameter MOUSEEVENTF WHEEL for a scroll command, or MOUSEEVENTF LEFTDOWN for a left button-pressed command, or MOUSEEVENTF_LEFTUP for a left button-released command, and MOUSEEVENTF RIGHTDOWN for a right button-pressed command, or MOUSEEVENTF_RIGHTUP for a right button-released command. As explained above, the value for "dwheel" may be fixed in some implementations (eg +120 for an upward scroll). In other implementations, this value may be encoded in the (eg.
parameterizable) movement sequence, e.g. as one or more fictitious movement values (dx, dy) (not shown). FIG. 46 is a variation of FIG. 45, wherein the two mouse devices M1, M2 are not connected directly to the computer 4601, but via a classic hub 4620. The special mouse filter driver 4663 and the special overlay application 4651 may be identical to those of FIG. 45.
FIG. 47 (a) to FIG. 47 (i) show some exemplary motion sequences that can be used by the mouse filter driver 4563, 4663, and that can be detected by the overlay application 4551, 4651 of the computer devices 4500, 4600, but of course the invention is not limited, and other suitable motion sequences can also be used.
Preferably, each predetermined sequence of motion messages is chosen such that the starting point and the end point remain the same, such that the native mouse cursor remains stationary for the user after scrolling the mouse, and / or after pressing or releasing the a push button, or after clicking or double-clicking, for example, the left mouse button and / or the right mouse button or another mouse button.
The exemplary series of FIG. 47 (a) to (f) each consist of five consecutive displacements, the exemplary sequences of FIG. 47 (g) and (h) each consist of six consecutive displacements, and the exemplary sequence of FIG. 47 (i) consists of seven consecutive moves, but sequences of less than five consecutive moves, or sequences of more than seven consecutive moves may also be used.
Preferably, each sequence contains at least two or at least three displacements different from the zero displacement, and preferably each sequence contains up to ten displacements.
In the exemplary sequences of FIG. 47 individual zero displacements (i.e., displacements with dx = 0 and dy = 0) are not used.
In preferred embodiments, each (e.g., predetermined) motion sequence includes a zero displacement (with dx = 0 and dy = 0) as the last individual motion. This displacement can be easily added to the examples of FIG. 47 (a) to (f). This has the advantage that the overlay application 4551, 4651 (in particular its message processor 4552, 4652) does not have to check for each incoming motion message whether one of the (e.g. predetermined) motion sequences occurs in the one or more buffers or text strings, but only need to do this when a motion message with the value dx = 0 and dy = 0 arrives. This is much faster and much more efficient.
Preferably, the cumulative displacement of each (e.g., predetermined) motion sequence is zero in the X direction and the Y direction.
In the examples of FIG. 47 only relatively small displacement values are used (e.g. with dx values and dy values in the range from -2 to +2), but larger displacements can also be used (e.g. with dx values and dy values in the range from -5 to +5 or in the range of -7 to +7, or even in the range of -10 to +10, or even more). The more individual motions each sequence contains, and / or the more back and forth motions, and / or the greater the displacement values, the more complex the sequence, and the less likely such sequence will ever appear "accidentally" as actual mouse motions. .
Since unwanted clicking (or dragging) can have greater consequences than unwanted scrolling (which is annoying, but can easily be undone by the user by scrolling back), it is best to select the sequence of the button-pressed message and the sequence of the message. make button-released message at least as complex as the sequence for the scroll-up message and the scroll-down message (e.g. more moves and / or a wider range of dx and dy values and / or more criss-cross moves, and / or zig-zag movements, and / or back and forth movements, instead of a smooth movement or a loop movement).
As stated above, in a simple implementation the mouse filter driver can convert any scroll-up message to the same (e.g. predetermined) motion sequence, regardless of the scroll value included in the message.
In another embodiment, for scroll movements, a value is included in the movement sequence, eg in a parameterizable sequence. This can be implemented, e.g., by extending the third or fourth predetermined set of moves (e.g., one of the sequences of FIG. 47 (a) to FIG. 47 (h)) with two additional moves, e.g., a move to right (e.g. with dx = the scroll value divided by 120; and with dy = 0), and an equal displacement back to the left, and only then the optional zero displacement described above. That way, the cumulative displacement still remains zero in both directions X and Y, and the scroll value indicated by the pointing device can be passed to the overlay application's message processor, which in turn can retrieve it from the sequence, and can forward to the operating system. Of course, an upward displacement and the same downward displacement will also result in a zero displacement. Tests have shown that providing the scroll value is not necessary for the intended desktop applications, it is sufficient to indicate only the direction (up or down), and therefore use two fixed sequences (without parameters).
FIG. 49 shows in more detail and with an example how the solution works with the motion sequences. M1 represents the first mouse device (eg the right mouse). M2 represents the second mouse device (eg the left mouse). "MFD *" represents the special mouse filter driver 4563, 4663. As above, many details are omitted and only the main principles of the invention are explained.
At low level, communication could be as follows: a pointing device is moved or operated (scroll wheel rotated or button pressed or button released), the pointing device communicates this via an interface (e.g. via a USB interface) to a HAL (hardware) abstraction layer), the HAL passes this on to a mouse device driver 4501, 4601 (also called "mouse driver" for short), the mouse driver passes this on to the mouse filter driver 4563, 4663, and the mouse filter driver passes this on to the operating system (eg via the Windows API in the example of Windows), and the operating system passes this on to the appropriate application 4562, 4662. More specifically, if the overlay application 4551, 4651 in click-through mode is configured, and / or if the native mouse cursor is over transparent pixels, the message is passed to an underlying application.
Referring to the example of FIG. 49, this could have the following functional operation: At time t1 the first mouse device M1 sends a motion message to the USB port (represented by "mov1" with displacement values dx and dy), this message goes to the Mouse Filter Driver which in principle just forwards the motion message to the operating system (symbolically represented by a cloud), possibly after format change or the like. As explained above (e.g. in FIG. 21) this message (or a copy or variant thereof, e.g. a message with the double displacement) will sooner or later also arrive in the message processor 4552, 4652 of the overlay application 4551, 4651 , where the message will be analyzed to update positions Pos1 and Pos2 and optionally adjust the position of the native mouse cursor 705 (e.g., as described above in FIG. 18a to FIG. 18c).
At time t2, the second mouse device M2 sends a scroll message scroll2 to the computer device 4501, 4601. The special Mouse Filter Driver 4563, 4663 receives this scroll2 message, but does not forward it as such to the operating system, but sends it in the place a sequence of (false or fictitious) move commands mov2a to mov2N to the control system, where N is the number of moves of the predetermined sequence (e.g., N = 5 for the sequences of FIGS 47a-47f, or N = 6 after addition of the zero displacement). The operating system forwards these (or at least some of the) movement commands to the respective underlying application as (real) movements of the native mouse cursor. In the example, it is the application located under the "first position" Pos1 (because the last movement was that of M1), which was actually not intended, because the second mouse device M2 sent the scroll message, not the first mouse device M1, and thus the scroll message should be sent from the second position Pos2. However, the (possibly wrong) underlying application (namely the application located at position Pos1) will process these (or some of these) move commands as harmless mouse movements, but will not execute a scroll command, which is one of the principles of the underlying invention.
Some time later (and the exact delay of this is of no importance) all these move messages also arrive at the overlay application's message processor, which will initially interpret the individual mouse movements as real mouse movements from the second mouse device M2, but when all motion messages have been processed, the overlay application (e.g. its message processor) will recognize the (e.g. predetermined) motion sequence, and as a result will send a scroll command to the O / S from the overlay application, namely from the second maintained position Pos2, as illustrated in FIG. 48.
The operating system will then pass this scroll command (which is sent from the overlay application) to the (correct) underlying application located "under" position Pos2, which application will execute this command by effectively scrolling the position Pos2 related to the right mouse, which was the intention.
It is a huge advantage that in this way it is possible to scroll simultaneously in two different documents (e.g. scroll down in a PDF document on the left side of the screen, and simultaneously, or almost simultaneously, scroll down or up in a text document located on the right side of the screen), without the user having to move the pointing devices, and without actually activating the underlying applications. This advantage should not be underestimated, because eg input from the keyboard is sent to the active application, which can remain active, while the user can still scroll in another document.
The attentive reader can ascertain when he reads FIG. 49 with FIG. 21 compares that the scroll message from mouse M2 is correctly processed as a scroll command from position Pos2, even though the native mouse cursor was on Pos1 when the mouse M2 was scrolled, without the user moving the mouse M2 manually move yourself first, in order to make the mouse cursor jump to Pos2 before scrolling. This is a huge advantage, because the user can now focus his attention on the task (e.g. reviewing a translation, or transcribing), and no longer has to wonder whether the mouse cursor is positioned correctly before scrolling or typing. click. In other words, thanks to the motion sequence solution, the two pointing devices (e.g. two mice) can be moved and / or operated very intuitively and without thinking. Tests have shown that even if the user is moving the other mouse meanwhile (in the example of FIG. 49: the right mouse M1), even then the scroll command (sent by M2) still reaches the correct underlying application which is located under Pos2.
This can be explained as follows. Suppose the user was moving the right mouse M1 at time t3. This (real) motion message is simply forwarded by the Mouse Filter Driver to the O / S, which in turn passes it on (possibly after adjustment, e.g. after doubling the dx and dy values) to an underlying application, namely that application where the native mouse cursor happens to be at that moment (usually Pos1 when M1 is moving). But since this is just a harmless mouse movement, it makes (for the intended uses)
doesn't really matter to which underlying application this message is sent. Since the native mouse position is put in the correct place (in the example of FIG. 49, on Pos2, because mouse M2 was scrolled) just before sending the effective scroll command (from the overlay application), it actually makes no matter where the native mouse cursor is when the scroll sequence arrives at the overlay application.
FIG. 49 thus schematically shows how a scroll message (scroll2) from the mouse M2 is eventually converted to a scroll command invoked from the second mouse position Pos2 from the overlay application, regardless of where the native mouse cursor is when M2 was scrolled.
Likewise, a "button pressed" message and a "button released" message from the first or second mouse device M1, M2 will be converted by the Mouse Filter Driver 4563, 4663 into a predetermined sequence. of “harmless mouse movements” that may be initially sent by the operating system to the “wrong” underlying application, (without causing any damage there), but which sequence of movements will be recognized by the overlay application some time later, and eventually will result in a button-pressed or button-released command that will be called from the correct position (i.e. the position related to the pointing device that was operated) from the overlay application, bringing the command to the correct underlying application even if the native mouse cursor was not in the position corresponding to the mouse whose button was pressed.
Above, the basics are explained to scroll and / or click buttons of two mice M1, M2 connected to the same computer, eg directly (FIG. 45) or indirectly via a classic hub (FIG. 46).
The mechanism with the predetermined (false) movement sequences also allows to correctly deal with certain "forbidden combinations", such as e.g. pressing a button of two different mouse devices simultaneously, and / or dragging one mouse simultaneously and scrolling with the other, or simultaneously pressing the left button and the right button of the same mouse device. Such prohibited combinations can either be completely captured in the mouse filter driver (and not in the overlay application), or completely in the overlay application (and not in the mouse filter driver), or partially in the mouse. filter driver and partly in the overlay application. In a preferred embodiment, as little intelligence as possible is put into the mouse filter driver, and as much as possible into the overlay application.
In preferred embodiments of the present invention, the message processing module 4552 and 4652 is further extended thereto to keep track of at least for the selected left and right mouse device whether a button has already been pressed, (or more specifically, whether the left button and / or the right button is pressed) and if so, no additional "button-
"pressed" command or scroll command to the operating system until the previously pressed button is released. The button status can be tracked, for example, by four bits, or by four Boolean variables, or other suitable means. In this way, the special overlay application 4551, 4651 and / or the special Mouse Filter Driver 4563, 4663 can ensure that the native mouse cursor can only be pressed and / or dragged and released at one location, even if the user would physically press a button on each of two different mouse devices at the same time An additional advantage of accommodating this in the overlay application is that the overlay application can optionally also display a visual text message or bitmap to the user to indicate that such action is prohibited.
FIG. 50 shows a flow chart of a computer-implemented method 5000 that can be executed by the special overlay application 4551, 4651 when executed on the computing device 4501, 4601 as shown in FIG. 45 and FIG. 46. At least a first pointing device M1, 4503R and a second pointing device M2, 4503L are connected to the computer device. The computing device 4501, 4601 further includes an operating system 4560, 4660 with a graphical user interface GUI that provides one native mouse cursor. The computing device 4501, 4601 of FIG. 45 and FIG. 46 further includes a Mouse Filter Driver 4563, 4663 which (as shown in FIG. 51) is configured: to receive motion messages from the first and second pointing devices 4503R, 4503L, and upon receipt of such message, the motion message or send a corresponding message (e.g. a message with the same meaning but in a different format) to the operating system; - at least one message from the group consisting of a button pressed message, a button released message, a scroll up message and a scroll down message from the first and / or the second pointing device 4503R, 4503L detect, and upon detection of such message, send a corresponding (e.g., a first, second, third, or fourth, respectively), e.g., predetermined motion sequence to the control system 4560, 4660. The computer-implemented method 5000, performed by the overlay application, includes the following steps: a) providing 5001 a graphic overlay window, optionally with one visible object, or with two visible objects, or with at least three visible objects. b) allowing 5002 or configuring the operating system O / S with the graphical user interface GUI to provide the native mouse cursor as a visible object; c) configuring 5003 the graphic overlay window in click-through mode (such as windows 584 and 884), or configuring the graphic overlay window as a substantially full-screen window consisting mainly or entirely of fully transparent pixels (excluding the optional movable objects); d) configuring 5004 of the operating system 4560; 4660 to send raw input messages related to the first and second pointing devices 4503R, 4503L, and providing a message processor 4552; 4652 to process these raw input messages; e) adjusting 5005 a first position or pointing position Pos1 in accordance with movements of the first pointing device 4503R, and adjusting a second position or pointing position Pos2 in accordance with movements of the second pointing device 4503L, f) adjusting 5006 a position of the native mouse cursor 4505, 4605 in accordance with the first or second tracked or maintained position Pos1, Pos2 depending on which pointing device was most recently moved and / or operated; g) detecting 5007 whether one of the (e.g., predetermined) motion sequences occurs in the stream of raw messages, and when such sequence is detected, determining which pointing device 4503R, 4503L caused the predetermined sequence; and - moving the native mouse cursor 4505, 4605 to the first or second location Pos1, Pos2 depending on which pointing device caused the sequence; and - sending a button-pressed command or a button-released command, or a scroll-up command or a scroll-down command from the overlay application 4551; 4651 to the operating system 4560; 4660, according to the detected motion sequence.
Preferably, steps e) to g) are performed repeatedly.
In one embodiment, step a) provides an overlay window that (in addition to the native mouse cursor provided by the O / S) includes at least one first visible object (e.g., in the form of an arrow), and step f) further comprises customizing 5006 from a position of that first visible object at the position (Pos2; Pos1) other than the position occupied by the native mouse cursor. In this embodiment, the user can see and move two movable objects: the native mouse cursor, and another object, e.g., a bitmap with a black arrow as shown in FIG. 8 or FIG. 9 or FIG. 10.
In another embodiment, in step a) a graphics overlay window is provided which contains at least a first visible object (e.g. in the form of a bitmap with an arrow or a hand) and a second visible object (e.g. also in the form of a bitmap). with an arrow or a hand); and step f) further comprises adjusting a position of said first visible object based on the first tracked or maintained position Pos1, and adjusting a position of the second visible object 706L based on the second tracked or maintained position Pos2. In this embodiment, the user can see and move two movable objects, e.g., two hands as shown in FIG. 34, or two lines as shown in FIG. 37, or two crosses as shown in FIG. 38.
FIG. 51 has already been discussed above.
Preliminary tests have shown that the mechanism with the mouse filter driver and the movement sequences allows, among other things, to: - move both mice at the same time, - keep one mouse stationary and scroll the other mouse (regardless of which mouse moves or operated the last one - hold one mouse stationary and click the other mouse (regardless of which mouse was last moved or operated), - move one mouse and click the other mouse at the same time, - keep both mice stationary, and both mice scrolling "simultaneously" (at least simultaneously for the user), moving both mice, and scrolling both mice "simultaneously" (at least simultaneously for the user).
This may not be the behavior that a typical game user wants, but is very convenient and very beneficial for professional computer users who want to use two mice, eg proofreaders when checking a translation; or transscribers who, for instance, activate and / or deactivate an audio player with the left mouse, and enter or edit text in a word processor with the right mouse and / or a keyboard and / or a speech recognition program.
In summary, in FIG. 45 to 51, therefore, proposed a solution where a pointing device (e.g., a mouse) displays a motion message, or a button-pressed message, or a button-released message, or a scroll-up message, or a scroll-down message. and where a mouse filter driver 4563, 4663 unconditionally converts the button pressed / released messages and the scroll up / down messages to a corresponding motion sequence which the overlay application 4551, 4651 converts back to corresponding commands.
This solution allows the user to work conveniently with two mice. For example, a transcriber can start / stop an audio player with one mouse and edit a text with the other mouse, and a translator can scroll in a PDF document with source text and type the translation into a text editor with the other mouse, and a reviewer can scroll through the source document with one hand and scroll through the target document with the other hand.
But the inventors faced additional problems. Tests showed that the mechanism described above does not allow clicking in applications such as.
"Task Manager" (English: "Task Manager"). Further investigation has shown that this problem appears to occur with applications running in elevated mode (also called "elevated mode").
A simple solution to this problem is to run the overlay application in "elevated mode" as well (eg by starting it "as Administrator"), but many of the targeted computer users do not have Administrator rights.
The inventors therefore wanted to find a different solution where the overlay application does not have to run in elevated mode.
They came up with the idea of providing a modified mouse filter driver that performs a method as depicted in the flow diagram of FIG. 52 (a). Comparison with the flow chart of FIG. 52 (a) and FIG. 51 teaches that steps 5202 to 5211 of FIG. 52 are identical to steps 5102 to 5111 of FIG. 51. Thus, the main differences are in steps 5201, 5212 and
5213.
In step 5201, the mouse filter driver receives a new message from a pointing device, and determines from which pointing device this new message originated, and determines the time when that new message was received, and remembers this information or some of this information, e.g. in RAM; At step 5212, the mouse filter driver checks if only one mouse has been used for at least a predetermined period (e.g., 100 ms); and if so, the mouse filter driver forwards the received message in step 5213 to the operating system without converting it to a motion sequence; and if so, the mouse filter driver performs one or more steps 5202 to 5211, checking in step 5202 whether the message is a move, and if so, the message is forwarded to the operating system, and if so, then converts the message to a corresponding (e.g., predetermined) motion sequence in a manner similar to that illustrated in FIG. 51.
Of course, further variations or refinements are possible. For example, step 5202 can be performed before step 5212, and the order of tests 5204, 5206, 5208, 5210 can be changed.
The predetermined period AT should be chosen sufficiently large for the overlay application to be able to jump the native mouse cursor. This predetermined period may be, for example, a period in the range of 20 to 500 ms, or 20 to 250 ms, or 40 to 200 ms, e.g. approximately equal to 50 ms or 60 ms or 70 ms or 80 ms or 90 ms or 100 ms, or approximately equal to 150 ms, or approximately equal to 200 ms, or approximately equal to 250 ms.
In other words, in the modified mouse filter driver, also referred to herein as the "conditional mouse filter driver", not every button-pressed / released message or scroll-up / down message is unconditionally converted to a motion sequence, but this conversion is Only applied under certain circumstances, eg when the user was "recently" working with another pointing device. The conversion is no longer applied as soon as the user works with only one pointing device for more than a predetermined time. Checking this can be implemented in different ways, e.g. by checking if the new message (received by the MFD) is from the same pointing device as the previous message (received by the MFD), and if the time difference between this new message and the first message from this pointing device, with no intermediate messages from other pointing devices, is greater than the predetermined period.
Variants of this are possible. FIG. 52 {b} shows a variation of the method of FIG. 52 (a), the step 5201 further comprising keeping track of whether at least one button of the pointing devices has been pressed (e.g., in a Boolean variable), and the method further comprising step 5214, testing whether at least one button has been pressed and the (new) message comes from another mouse, and if so, the message is ignored in step 5215, if so, then proceeds to step 5212.
Steps 5202 through 5211 of FIG. 52 (a) are briefly represented by the block
5220.
Thus, step 5214 acts as an additional filter in the event that at least one button is pressed. If none of the buttons are pressed (which in the example of FIG. 45 means that none of the buttons M2B1, M2B2, M1B1, M1B2 are pressed), then the message will continue to be handled as shown in FIG. 52 (a).
Tracking whether a button has been pressed can also be implemented in another way, e.g., by means of a counter, which is incremented by one for each button-pressed message, and decremented by one for each button-released message. Testing whether at least one button has been pressed can then be done by testing whether this counter has a value greater than zero. But the invention is not limited to this, and other similar ways can be used as well.
FIG. 52 (c) shows, by way of an example, how a mouse filter driver (MFD) that uses the method of FIG. 52 (a) or FIG. 52 (b) processes messages from mouse M1 and mouse M2.
At time t1, mouse M1 sends a movement message. This is simply passed on. At time t2, mouse M1 sends a movement message. This is simply passed on. At time t3, mouse M2 sends a movement message. This is simply passed on. At time t4, mouse M1 sends a scroll message. This is translated by the MFD into a motion sequence regardless of the time difference t4-t3 because the previous message came from mouse M2.
At time t5, mouse M2 sends a scroll message. This is translated by the MFD into a motion sequence regardless of the time difference t5-t4 because the previous message came from mouse M1.
At time t6, mouse M2 sends another scroll message. This is simply passed on, because the mouse M2 has been active for longer than a predetermined time (eg more than 100 ms) and no messages from M1 came in between. Thus, it is not necessary to use a motion sequence, because it can be assumed that the native mouse cursor is already correct at time t6.
At time t7, mouse M2 sends a button-pressed message (press). This is simply passed on, because the message comes from the same mouse M2, which has been active for longer than the predetermined period without messages from the other mouse M1.
At time t8, mouse M2 sends a button-released message (release). This is simply passed on.
It is noted that at least one button has been pressed in the period from t7 to t8, but since mouse M1 does not send anything during this period, no messages need to be ignored.
At time t9, mouse M1 sends a button-pressed message (press). This is converted to a motion sequence by the MFD regardless of the time difference t9-t8 because the previous message came from mouse M2. The variable that keeps track of whether a button has been pressed is TRUE, or the counter is 1.
At time t10, mouse M2 sends a movement message. This message is ignored because a button has been pressed, and the message from M2 does not come from the same mouse as the one whose button was pressed.
At time t11, mouse M1 sends a button released message. This message is converted to a motion sequence by the MFD, because in the example of FIG. 52 (c) the time difference t11-t9 is less than the predetermined period ΔT.
It is noted that if an application in elevated mode (eg Task Manager) was clicked at time t7, this clicking would work effectively regardless of whether the overlay application is running in elevated mode or not. However, if Task Manager was clicked at time t9, this would not work unless the overlay application was started as Administrator.
FIG. 52 (d) shows by means of a second example how a mouse filter driver (MFD) that uses the method of FIG. 52 (a) or FIG. 52 (b) processes messages from mouse M1 and mouse M2.
At time t1, mouse M2 sends a movement message. This is simply passed on.
At time t2, mouse M2 sends a scroll message. This is also simply passed on, because (in this example) the time difference t2-t1 is greater than the predetermined period AT.
At time t3, mouse M2 sends a button-pressed message. This message is not converted to a motion sequence, but is simply passed, because t3-t1 is greater than AT. At least one button has now been pressed.
At time t4, mouse M1 sends a scroll message. This message is ignored because at least one button has been pressed. Since this message is ignored, and the native mouse cursor will not change position, the time from the first message from T2 (at t1) can continue as usual.
At time t5, mouse M2 sends a button released message. This is passed simply because £ 5-t1 is greater than AT, and the message from t4 was ignored, at least in the method of FIG. 52b.
At time t6, mouse M1 sends a button-pressed message. This message is converted to a motion sequence because the previous message was from another mouse (namely M2).
At time t7, mouse M2 sends a movement message. This message is ignored because a button of mouse M1 has been pressed.
At time t8, mouse M1 sends a movement message. This is simply passed, because in the example t8-t6 is greater than AT, and the message from t7 was ignored.
At time t9, mouse M2 sends a scroll message. This message is ignored because a button of mouse M1 is still pressed.
At times t10 and t11, mouse M1 sends another movement message. These are simply passed, because t10-t6 and t11-t6 are greater than AT, and the message from t9 was ignored.
At time t12, mouse M2 sends a scroll message. This message is ignored because a button of mouse M1 is still pressed.
At time t13, mouse M1 sends a button released message. This message is simply passed on by the MFD. From now on no button has been pressed.
At time t14, mouse M2 sends a scroll message. This message is converted to a motion sequence regardless of the time difference t14-t13, because the previous message came from the other mouse M1.
In a variant of this method, when a button-pressed message "as is" is sent to the O / S (i.e. not via a motion sequence), the button-released message "as is" is also sent to the O / S. .
In a variation of this method, if a button-pressed message was treated "as a motion sequence" (e.g., at time t6 of FIG. 52d), the button-released message is also treated as a motion sequence. In the example of FIG. 52d the button release on t13 would therefore also be sent as a movement sequence. In this way, the button press and button release can exhibit a more consistent behavior, eg when clicking on applications in Evolved mode.
Tests have shown that an overlay application with the "modified mouse filter driver" (executing the method of FIG. 52a or FIG. 52b) works very well, effectively allowing a computer user without Administrator rights to click on an application running in powered mode (such as Task Manager), provided the user has the other pointing devices (e.g., the mouse in the non-
dominant hand) does not move or operate for at least the predetermined period. The tests show that although the user sometimes has to click multiple times, if he / she was moving both pointing devices the first time, when the user notices that "the clicking is not working", then his / her spontaneous response is that he / she only will operate the pointing device of the dominant hand, after which the clicking will work.
Further or other variants are possible.
In a variation (not shown) of FIG. 52 (b) the message from the other mouse (e.g., motion message or scroll message or button message) is not simply ignored in step 5215, but a zero displacement message is sent to the O / S. This message will eventually arrive at the overlay application that has been informed that a message from the other mouse has been ignored, whereupon the overlay application can e.g. display a visual message on the screen at the position of this other mouse, (e.g. in the form of a red letter X, or a red circle with a dash, or the like), to indicate to the user that the respective movement or click or scroll action from this mouse is prohibited.
FIG. 52 (e) shows another or further variation of FIG. 52 (b), also verifying in step 5214 whether a button has been pressed, and whether the message is from a different pointing device (e.g. mouse) than the one whose button was pressed, but where (if the result of this test is true) the message is not simply ignored (as in FIG. 52b), but it is further checked in step 5231 whether the message is a button message or a scroll message (e.g., a button-pressed message or a button released message or a scroll-up message or a scroll-down message), in which case the message is indeed ignored (in step 5232), but if the message is a motion message (checked in step 5233), then the displacement values dx and dy in the message are adjusted, after which the message is indeed sent to the operating system (step 5234).
Adjusting the displacement values may include, e.g., scaling or limiting these values such that the dx values and dy values fall, e.g., within a predetermined range, e.g., from -5 to +5, or from -4 to +4, or from -2 to +2, or from -1 to +1. Tests have shown that it can sometimes be useful that the position of the visible object related to the left mouse can be moved, while eg with the right mouse a certain text fragment is selected in a text document. These tests show that when the displacement values dx, dy of the movements are greater than a certain value (in the tests a value of about 10 or 15), the selected text fragment may temporarily jump, which is confusing for the user. But when the displacement values dx, dy are scaled or limited to a predetermined range, as in step 5234, the text fragment jumps less often. This is useful, for example, for users who want to select a particular text fragment in a target text with the mouse in the dominant hand, while moving the mouse in the non-dominant hand to underline a corresponding text fragment in the source document. (for the sake of completeness: the tests show that choosing the starting point between two text lines is more sensitive than choosing a starting point that lies on a text line).
Further variants are possible.
For example, instead of merely ignoring a button message in step 5215 or 5232, it can be tracked that a button-pressed message from the other mouse device has been blocked, and the message processor will later also release the corresponding button-press. withhold message. In this way, more consistent behavior can be achieved.
For example, in a further or different variation, the movement of the other pointing device in step 5234 can be forwarded to the control system as a movement sequence, with or without scaling or limiting the displacement values. Such a movement sequence can be defined, for example, as a movement sequence composed of a partial sequence according to one of the sequences shown in FIG. 47, followed by the displacement message with values (+ dx, + dy), followed by an opposite displacement message with values (-dx, -dy), preferably further followed by a zero displacement (with dx = 0, dy = 0). The overlay application will receive and recognize this movement sequence, and can deduce from it what movement the other pointing device made, and can eg decide to update the position of the associated visible object, but not to jump the native mouse cursor.
But the invention is not limited to this, and other variants are possible.
The inventors realize that the conversion to motion sequences can also be achieved without using a mouse filter driver, but by using special pointing devices (see FIG. 53) or a special hub (see FIG. 54).
FIG. 53 shows a computer system 5300, which can be viewed as a variation of the computer system 4500 of FIG. 45, with the main difference that the computer system 5300 does not include a mouse filter driver 4563 as described above, but does include two special pointing devices M1 ** and M2 **, e.g. a left mouse 5303L and a right mouse 5303R. These mice can have the same hardware configuration as shown in FIG. 23 (b), but the controller 2324 will function differently. More specifically, the control unit will be provided to send a movement message when the mouse is moved, and to send a first (e.g. predetermined) movement sequence when a left button is pressed, and a second (e.g., predetermined) movement sequence when a right button is pressed. predetermined) motion sequence, and to send a third (fixed or parameterizable) motion sequence when scrolling up the scroll wheel, and to send a fourth (fixed or parameterizable) motion sequence when scrolling down the scroll wheel, on a similar manner as described in FIG. 45 to FIG. 51, but now from the pointing device itself.
Preferably in this case the overlay application runs in elevated mode, otherwise the user cannot control certain applications running in elevated mode (eg Task Manager) with such mouse, but this is not absolutely necessary.
Optionally, the special mouse 5303L, 5303R is equipped with two operating modes, whereby the operating mode can be set eg by the user, eg by means of a hardware switch, or a push button, or in software by a setting program. The two modes can include a classic mode where button and scroll messages are forwarded to the computer as such, and a special mode (also called "dual mouse mode") where the mouse will transmit the motion sequences described above.
FIG. 54 depicts a computer system 5400, which can be viewed as a variation of the computer system 4600 of FIG. 46, the main difference being that the computer system 5400 does not include a mouse filter driver 4663, but a dedicated hub 5420.
In one embodiment, the hub is provided to perform a method similar to the method 5100 shown in FIG. 51 (wherein all click messages and scroll messages from the pointing devices M1, M2 are unconditionally converted to motion sequences), except that the messages or the motion sequences are sent to an input port of the computer, e.g. a USB port.
In one embodiment, the hub 5420 is provided to perform a method similar to the method 5200 shown in FIG. 52, where not all click messages and scroll messages are converted to motion sequences, but only under certain conditions, as described above at step 5212 of FIG. 52.
Optionally, the special hub 5420 is equipped with two operating modes, where the operating mode can be set eg by the user, eg by means of a hardware switch, or a push button, or in software by a setting program. The two modes may include a classic mode where commands from multiple mice are forwarded to the computer as such, and a special mode (also referred to as "dual mouse mode") where the hub is configured to follow the method 5100 of FIG. 51, or the method 5200 of FIG. 52 to perform.
For example, the dedicated hub 5420 may have the same hardware configuration as the hubs shown in FIG. 27 (a) or FIG. 27 (b), the controller 2744 or 2745 being provided to perform the method of FIG. 51 or FIG. 52.
After the experiences with the mouse filter driver, the inventors realized that it must be technically possible for manufacturers of existing operating systems to expand the existing operating system to a version showing two visible mouse pointers, which can be individually moved by two indicators, where the two indicators may be moved simultaneously, and where the two indicators (from a user's point of view) may be scrolled at the same time, and where one indicator may be scrolled while the other is being moved, and each of the indicators (separately) can be clicked or dragged, but not simultaneously, as shown in the following table, for example: pee [ia qE pee "eq ee CC Ane ene Table 1: simultaneous actions with first and second pointing device By excluding the two pointing devices being clicked simultaneously or dragged, only one application "ac be "active" (as is now the case), and the operating system can continue to function essentially as it does now, with the difference that instead of one visible object moving with a mouse (called "mouse cursor}), it now displays two mouse cursors .
In certain embodiments, while the operating system displays two visible movable objects, there is actually still only a single "native mouse cursor" underlying it. This native mouse cursor can then be shown eg as a white arrow, but preferably the bitmap of this mouse cursor can change depending on where it is (eg above a visible component). The other mouse cursor could not be a real mouse cursor, but just an image, eg a black arrow or a blue arrow or a red arrow, that is being moved.
Since there is actually still only one native mouse cursor, such an operating system can be made 100% backward compatible with existing operating systems. By swapping the active and passive mouse cursor (or moving the native mouse cursor) (or swapping), the user can apparently still work with two pointing devices, even though not all combinations are allowed (see e.g. Table 1) .
But the inventors went a step further and realized that under certain circumstances it must also be possible to allow scenarios (v) and (vi) anyway.
Both embodiments are described in more detail below.
Providing two mouse cursors by the operating system offers several advantages over an implementation with an overlay application and a mouse filter driver, including: - the two mouse cursors can both have a Z-order higher than the other applications (see FIG 55c); - since the operating system has full control over which messages are sent as commands or events to underlying applications, the operating system does not need to build in any additional delay as described e.g. in FIG. 22 and FIG. 26, and the control system does not need to use (e.g., predetermined) motion sequences to avoid certain "racing conditions". In other words, with proper provisioning, a button-press message sent by the right mouse will not trigger a button-press command at the position following movements of the left mouse; - the operating system can handle certain tasks in Kernel mode, which an overlay application, for example, cannot; - the operating system has access to more system resources, and has certain knowledge that a mouse filter driver, for example, does not have, such as user settings in terms of screen resolution and / or mouse speed. For example, the operating system can quite easily limit the movements of the mouse cursors at the edges of the screen.
Such an operating system would solve the problems described in FIG. 1 to FIG. 4 can be solved in an elegant way as well, without the need for a mouse filter driver, and without the need for a dedicated mouse or hub, and without an overlay application.
Since this need exists, but such an operating system does not yet exist, it can hardly be otherwise than that there is a prejudice that it is not technically possible to provide such functionality in a technically feasible way. Perhaps the manufacturers of existing operating systems are blind to supporting the mice of multiple users connected to one computer via a network, and they forget that it would be an added value if the same user could use both hands. Or they may be put off by the notion that when two mice are connected to the same computer, all possible combinations of move and click and drag and scroll should be supported (which is not necessary). Or perhaps the manufacturers of existing operating systems think that using multiple mice is only useful for gamers, but not for professional users. The examples of FIG. 1 to FIG. 4 clearly demonstrate, however, that there is an added value for such an operating system, e.g. with professional computer users such as translators, proofreaders, transcribers, technical writers, lawyers, etc. Anyway, the fact is that (as far as the inventors know ) such an operating system does not yet exist.
FIG. 55 (a) and FIG. 55 (b) show a block diagram of a computer system 5500 according to an embodiment of the present invention. The shown in FIG. 55 (a) computer system 5500 shown includes a computer device 5501 connected to a first and second pointing devices 5503R, 5503L, and connected to a display 5504, and to a keyboard 5502.
The computing device 5501 includes a control system 5560 (see FIG. 55d) that provides two visible objects 55054, 5505p that track movements of the pointing devices 5503R, 5503L.
As explained above (at FIG. 28a) it is possible (in a first user mode, e.g. called "right-handed mode") to have the active mouse cursor always follow movements of the first pointing device (e.g. the right mouse, by analogy with the RightH mode described in Fig. 28a), and have the passive mouse cursor always follow movements of the second pointing device (e.g., the left mouse). In this mode, the native mouse cursor will follow the movements of the first mouse, not jump. There is therefore also no clicking or scrolling with the second pointing device. And should this pointing device be clicked or scrolled anyway, these messages can be blocked.
It is also possible to provide a second user mode, (eg called "left-handed mode") where the active mouse cursor always follows movements of the second pointing device (eg the left mouse), (analogous to the LinkH mode described in FIG. 28a ), and the passive mouse cursor always follows movements of the first pointing device In this case, the first pointing device cannot click or scroll, and should this pointing device be clicked or scrolled, these messages can be blocked.
But the most interesting embodiment is, of course, the one where the user can move and / or scroll and / or click and / or drag both pointing devices, analogous to the TwoH mode described in FIG. 28a. In what follows, especially the two-handed mode is further explained.
FIG. 55 (a) shows movements of the first mouse cursor 5505a in accordance with movements of the first pointing device 5503R, e.g., a right mouse.
FIG. 55 {b} shows movements of the second mouse cursor 5506L in accordance with movements of the second pointing device 5503L, e.g., a left mouse. The left mouse and right mouse can be identical mice (in terms of hardware and / or firmware), or can be different mice, eg from different manufacturers.
In the example of FIG. 55 (a) and FIG. 55 (b) the shape and size of the first mouse cursor 5505a is the same as that of the second mouse cursor 5505p and only the color is different, but it is not necessary. As already stated, in preferred embodiments, the appearance of the first and / or the second mouse cursor may vary depending on which visual objects are below the respective mouse cursor. For example, the mouse cursor 5505a can take the form of a bitmap with a "hand" when an underlying visual object can be dragged, or take the shape of an "I" (so-called cross-hair) e.g. when the respective mouse cursor is in a text field, or take the form of an East-West arrow when a visual object can be scrolled left-right, etc. This is a very important advantage (compared to objects that cannot change shape), as it provides the user with visual information provides information about possible actions, and provides information whether the cursor is positioned correctly (eg when the user wants to reduce the width or height of an application window). FIG. 55 (c) is a schematic representation of a so-called "Z-order" of windows or image planes and a "cursor plane" 5586 as may be used in the computer system 5500 of FIG. 55 (a, b). In the example there is a bottom plane 5581, herein also referred to as "desktop" or "desktop window" located at height Z1. At height Z2 is a window 5582 associated with a first application 55624, eg a text editor. At height Z3 is a second window 5583 associated with a second application 5562b, e.g., a PDF viewer At height Z4 is the so-called "cursor plane" which in this example contains two separate bitmaps, one for the first mouse cursor 5505a (herein also "active mouse cursor" referred to as), and one for the second mouse cursor 5505p (also referred to herein as "passive mouse cursor"). These designations are intended to simplify the description of the invention. The active and passive mouse cursor can be located anywhere on the screen, e.g., left or right. , or yourself are on different screens.
Of course, the representation of FIG. 55 (c) just an example, and many variants are possible. For example, there may be more than two, or less than two, user applications running in the computing device, and / or windows 5582 and 5583 may overlap. It is also possible to place the two mouse cursors 55054, 5505p in separate cursor areas.
FIG. 55 {d} shows a simplified high-level block diagram of hardware and software components located in the computer system of FIG. 55 (a, b), wherein the computer system 5500 includes a computer device 5501 with a special operating system 5560 * that provides two mouse cursors 55054, 5505p, which can be moved separately by two pointing devices 5503R, 5503L. These pointing devices can be classic pointing devices, e.g.
a mouse, a touch pad, a track ball, a stylus, etc. In a preferred embodiment, the two pointing devices are HID compatible mice M1, M2, each with at least two or exactly two push buttons (M1B1, M1B2) and (M2B1, M2B2), and preferably also at least one or exactly one scroll wheel (not shown).
In the example of FIG. 55 (d) the two mice M1, M2 are directly connected to the computer, e.g. via a USB port, but this is not necessary, and the two mice can also be connected to the computer via a classic hub (not shown, but see, e.g., FIG. 12).
In contrast to the block diagram of FIG. 5 (f), FIG. 11, FIG. 12, FIG. 23a, FIG. 24, FIG. 25, FIG. 45 and FIG. 46, the block diagram of FIG. 54 no dedicated overlay application, but the two mouse cursors are provided by the operating system itself. While of course many implementations are possible, this embodiment basically means that the operating system includes one or more software modules or routine that perform the same or similar functions of the aforementioned "message processor" 5352 and the "repositioning routine" 5353 of e.g. FIG. 53. In FIG. 54 this is represented by one module 5562 * called "message processor and cursor positioning". It maintains the first position PosR with coordinates (x1, y1) according to movements of the first pointing device 5503R, and the second position PosL with coordinates (x2, y2) according to movements of the second pointing device 5503L, according to the respective movement messages. These messages can be delivered e.g. via a USB port, a hardware abstraction layer (HAL), one or more mouse device drivers 5561, etc.
FIG. 56 depicts a high-level flow diagram of an exemplary computer-implemented method 5600 that can be performed by the dedicated operating system 5560 *, e.g., by the exemplary "message processor and cursor positioning" 5562 * shown in FIG. 55 (d). The 5600 method shown in FIG. 56 comprises the following steps: b) providing 5602 an operating system 5560 * with a graphical user interface, with a first and second pointer positions PosR (x1, y1), PosL (x2, y2), and with a first mouse cursor 5505a (also called active mouse cursor) and a second mouse cursor 5505p (also called passive mouse cursor); c) receiving 5603 an input message from the first and / or the second pointing device 5503R, 5503L; d) updating the first and second positions (PosR, PosL) in accordance with movements of the first and second pointing devices (5503R, 5503L); e) adjusting the position Posa of the active mouse cursor 5505a and the position Posp of the passive mouse cursor 5005p based on which pointing device 5503R, 5503L was last moved or operated; f) further processing the input message, e.g. by sending or not sending a command or event to an underlying visual component or application.
It is noted that step d) does not, but step e) does depend on the said LeftH, RightH, TwoH mode]. (in the example of FIG. 56, in step e) the TWO mode is used).
FIG. 57a shows in more detail what the processing of the input message (step 5607) might look like. In the method of FIG. 57a, the method further comprises keeping a status of one or more or all of the buttons of the pointing devices (e.g., pressed or not pressed), and is taken into account for the handling of subsequent messages.
In step 5701 it is checked whether a button has been pressed. The state of the buttons M1B1, M1B2, M2B1, M2B2 of the first and second indicating device M1, M2 can be kept for example by means of four bits, or by means of four boolean variables, or by means of a counter which keeps or at least one button is pressed, as described above.
If a message comes in and a button was pressed (Y), then it is checked in step 5712 whether the incoming message is from the same pointing device as the pointing device from which a button was pressed. And if the result of this test is true (Y), then proceeds to step 5704; and if the result of this test is false (N), then the message is ignored (step 5713) and optionally an error message is displayed.
If it was determined in step 5701 that no button was pressed, then proceed to step 5704.
In step 5704, it is checked whether the received message was a button-pressed message, and if so, the button-pressed message is treated in step 5705 as a "button-pressed command", e.g., by a "button-pressed" event "to the component or application located at position PosL or PosR depending on which pointing device was pressed, and the status of the buttons is updated, in step 5706 it is checked whether the received message was a button released message, and if so then the button released message in step 5707 is treated as a "button released command", e.g. by sending a "button released event" to the component or application located at position PosR or PosL depending on which pointing device was pressed, and the status of the buttons is updated, in step 5708 it is checked whether the received message was a scroll-up message, and if so, the scroll-up message in step 5709 is treated as a "scroll-up command ", eg. by sending a "scroll-up event" to the component or application located under PosR or PosL depending on which pointing device the message originated from; otherwise (i.e. no scroll-up message) it is checked in step 5710 whether the received message was a scroll-down message, and if so, the scroll-down message is treated in step 5711 as a "scroll-down command ", e.g. by sending a" scroll-down event "to the component or application located under PosR or PosL depending on which pointing device the message originated from.
Many variations are of course possible. For example, the order of the tests can be changed, and / or the messages can be processed in a different way, e.g. by function calls, etc.
In the method of FIG. 57a, thus, when a button of one pointing device is pressed, other messages are ignored, but messages coming from the same pointing device are handled.
In a variant of this method (not shown), motion messages are further processed (this is "dragging"), and a button-released message, but other messages are blocked, in particular a scroll message or a button-pressed message. message related to another button of the same pointing device, according to scenario (i) to (iv) of Table 1.
FIG. 57b shows a variation of the method of FIG. 57a, wherein it is checked in step 5712 whether the message is a motion message or a button released message from the same pointing device as the one from which a button was pressed, and if so, the message is further processed in steps 5704 to 5711 as above, and if not, the message is ignored in step 5713, and an error message is optionally displayed, e.g., in the form of a bitmap.
In another or further variant of this method (not shown), a message from another pointing device when a button has already been pressed is not merely ignored, but a record is kept in step 5713 that a button-pressed message has been ignored, and is step 5707 performs an additional test to see if the associated button pressed message was ignored, and if so, the button released message is also ignored. In this way it can be prevented that a button-released command or event is sent to a component or application that is not preceded by a button-pressed command.
The methods described above try to keep the operating system's behavior as retro-compatible as possible, so that a majority of existing applications will continue to run smoothly, without any changes to the application.
But the inventors came up with the idea that it should be technically possible to provide additional functionality beyond what is currently possible with existing operating systems.
For example, it should be technically possible (at least from an application point of view) to provide an operating system that allows selecting text with the right mouse button in a first application (e.g. in a text editor on the right of the screen), while simultaneously scrolling in a PDF document viewer with the left mouse (or vice versa). This is currently not possible with an operating system that has only one mouse cursor, as the mouse cursor would have to move from one application to another in order to scroll, but such movement is not readily possible while selecting text, or during the dragging an object, see scenario (v) and (vi) of Table 1. However, such functionality can be provided in an improved operating system, because there is basically no reason why the text application (on the right side of the screen) should know that scrolling is in progress in another application (eg on the left side of the screen, or on another screen), especially when using a different mouse cursor.
FIG. 58 shows a flow chart of a method 5800 that can be performed by such a control system. This method 5800 can be viewed as a variation of the method 5600 of FIG. 56, the main differences being: that in step 5805 it is further determined which (e.g. visual) component (s) or application (s) are located at the first and second designation position PosR, pol; that the processing of the message in step 5807 takes into account which (e.g. visual) component (s) or application (s) are under the first and second pointer position PosR, PosL.
In an exemplary embodiment, the control system checks whether the first pointer position PosR and the second pointer position PosL are over the same visual component and / or over the same application window, and the further processing of the input message in step 5607 takes this into account.
In a specific embodiment, if the two mouse cursors are over the same visual component or over the same application window, an error message is displayed in at least one of the following cases: * upon receipt of a button-pressed message from the first pointing device (5503R) if a button (M2B1, M2B2) of the second pointing device (5503L) has already been pressed; * upon receipt of a button pressed message from the second pointing device (5503L) if a button (M1B1, M1B2) of the first pointing device (5503R) has already been pressed; * upon receipt of a scroll-up message or a scroll-down message from the first pointing device (5503R) if a button (M2B1, M2B2) of the second pointing device (5503L) has already been pressed; * when receiving a scroll up message or a scroll down message from the second pointing device (5503L) if a button (M1B1, M1B2) of the first pointing device (5503R) has already been pressed.
Preferably, in these cases, no corresponding command or event is sent to the underlying visual component or application, so the component or application does not need to know that a button or scroll wheel of another pointing device has been operated. So for that application, it looks like the operating system has only one pointing device.
In a further or different embodiment, if the first and second mouse cursor (55054, 5505p) are over different visual components or application windows: * upon receipt of a scroll-up message from the first pointing device (5503R), a "scroll up "command or event sent to the underlying component or application (e.g., from the first pointing position), even if a button (M2B1, M2B2) of the second pointing device (5503L) has already been pressed; * upon receipt of a scroll down message from the second pointing device (5503L), a "scroll down" command or event is sent to the underlying component or application (e.g. from the second pointing position), even if a button (M1B1, M1B2) of the first indicating device (5503R) has already been pressed.
Thus, in this embodiment, simultaneous dragging with one mouse and scrolling with the other mouse is allowed, provided that the first mouse cursor and the second mouse cursor are over different visual components or over different applications, as shown in Table 2.
First Second Accepted and | Teen [an ES De d "= e ES [ee jee + [jen eee ES (v) | click, drag, scroll no, if above the same component or application; yes, if above several components or applications (vi) | click, drag click, drag no, if over the same component or application; yes, if over different components or applications Table 2: Simultaneous actions with first and second pointing device in an enhanced O / S FIG.59a and FIG.59b show a computer system 5900 that is a variant is from the computer system 5500 of Fig. 55a and Fig. 55b The main difference is that the computer system 5800 of Fig. 58 includes a control system that provides three visible objects 5905pL, 5905R, 5905n that selectively move along with one of the pointing devices. In the example of Fig. 58, the three visible objects comprise two bitmaps with a black arrow and one bitmap with a white arrow, with the white arrow located above the black arrows.
In FIG. 59 (b) the first pointing device is moved.
In FIG. 59 (b) the second pointing device is moved.
FIG. 59 (c) is a schematic representation of a so-called "Z-order" of windows or image planes and a "cursor plane" as may be used in the computer system of FIG. 59 (a). This needs no further explanation.
In the two-handed mode TweeH, the two black arrows 5905L, 5905R follow movements of the indicating devices M1, M2 at all times. These bitmaps are thus mainly located at the aforementioned pointer positions posR, posL (also called first and second position). The bitmap with the white cursor is the "native mouse cursor", and in two-handed mode TwoH it jumps from one position to another depending on which pointing device was last moved or operated.
The two objects can also be called "mouse pointers", but to differentiate from the native mouse cursor they may be better called "passive mouse pointers".
An advantage of this embodiment with three visible and movable objects or mouse pointers or mouse cursors is that the user sees the two pointer positions at all times (in the example above: the black arrows), and that he also sees clearly where the "native mouse cursor" is located.
He will also see that the native mouse cursor does not jump when eg the left button M1B1 of the right mouse M1 is pressed, eg while dragging with the right mouse, and will understand that it is apparently not allowed to click simultaneously with the left mouse. Optionally, an error message can also be displayed (as explained above), but this is not strictly necessary.
But of course the invention is not limited to three bitmaps including the native mouse cursor, which can change shape depending on the underlying component or application, and two bitmaps each with a black arrow, but of course also a bitmap with a blue or red arrow. or an object of a different shape, e.g. a hand, and it is also possible that these two bitmaps also change shape depending on the underlying component or application.
In one embodiment, the native mouse cursor is and the two other bitmaps are opaque, but these two other bitmaps can also be semi-transparent.
But other representations are of course also possible.
FIG. 59 (d) and FIG. 59 (e) show two bitmaps that can also be used.
Finally, although individual features have been explained in different drawings and in different embodiments of the present invention, it is possible for features of different embodiments to be combined, as would be logical to those of skill in the art reading this document.
While the present invention is explained for systems having only two pointing devices, the present invention will also work with more than two pointing devices connected to the same computer, for example with three pointing devices.

权利要求:
Claims (23)
[1]
A computer-implemented method (5000) for supporting two pointing devices in a computer device (4501) to which a first pointing device (4503R) and a second pointing device (4503L) are connected, and operating an operating system (4560) with a graphical user interface wherein the operating system provides a native mouse cursor, the method comprising the steps of: b) allowing or configuring (5002) the operating system (4560) with the graphical user interface to provide the native mouse cursor as a visible object (705); d) providing a message processor (4552; 4652) to process raw input messages, and configuring (5004) the control system (4560; 4660) to output raw input messages from the first and second pointing devices (4503R, 4503L). send to this message processor; e) maintaining and / or adjusting (5005) a first position (Pos1) in accordance with movements of the first pointing device (4503R), and adjusting a second position (Pos2) in accordance with movements of the second pointing device (4503L ); f) adjusting (5006) a position of the native mouse cursor (705) in accordance with movements of the first or second pointing device (4503R, 4503L) that were most recently moved and / or operated; g) detecting (5007) whether the raw input messages form or include a motion sequence according to a predetermined format, and upon detection of such motion sequence: - determining which pointing device (4503R, 4503L) caused this motion sequence; and - moving the native mouse cursor (705) to the tracked or adjusted position (Pos1i, Pos2) associated with the pointing device (4503R, 4503L) that caused the motion sequence; and - sending at least one command selected from the group consisting of a button-pressed command, a button-released command, a scroll-up command and a scroll-down command to the operating system (4560; 4660) according to the detected motion sequence.
[2]
A computer-implemented method (5000) according to claim 1, further comprising the step of: a) providing (5001) a graphics overlay window (584).
[3]
A computer-implemented method (5000) according to claim 2, wherein in step a) a graphics overlay window (584) is provided as a substantially full screen window consisting mainly of fully transparent pixels;
and / or wherein the method further comprises the step of: c) configuring (5003) the graphics overlay window (584) in click-through mode.
[4]
A computer-implemented method (5000) according to claim 2 or 3, wherein in step a) a graphics overlay window is provided comprising at least one visible object (706R; 806; 906; 1006); and wherein step f) further comprises adjusting a position of said visible object at the position (Pos2; Pos1) from the first and second positions (Pos1, Pos2) other than the position occupied by the native mouse cursor.
[5]
A computer-implemented method (5000) according to claim 2 or 3, wherein in step a) a graphics overlay window is provided comprising at least a first visible object (706R; 806; 906; 1006) and a second visible object (706L) ; and wherein step f) further comprises adjusting a position of the first visible object (706R) based on the first position (Pos1), and adjusting a position of the second visible object (706L) based on the second position. (Pos2).
[6]
A computer-implemented method (5000) according to any preceding claim, wherein each of the motion sequences comprises a series of two to twenty movements, or comprises a series of three to ten movements.
[7]
A computer-implemented method according to any preceding claim, wherein a cumulative move of the motion sequences is a zero move.
[8]
A computer-implemented method according to any one of the preceding claims, wherein each of the motion sequences last displacement comprises a zero displacement (dx = 0; dy = 0).
[9]
A computer-implemented method according to any preceding claim, wherein step e) comprises adjusting the first position (Pos1) and the second position (Pos2) based on the raw input messages.
[10]
A computer-implemented method according to any preceding claim, wherein the operating system is Windows 7 or Windows 10 or a later Windows version of Microsoft Corporation,
or where the operating system is mac OS Sierra or mac OS High Sierra or a later mac OS version from Apple inc, or where the operating system is Linux.
[11]
A computer program product containing executable instructions that performs the method of any preceding claim when executed by a computer device (4501; 4601) to which a first and second pointing device are connected.
[12]
A computer-implemented method (5100; 52004; 5200b) for filtering messages from at least two pointing devices (4503L, 4503R; 4603L, 4603R), the method comprising the steps of: a) receiving (5101; 5201) of a message from one of the pointing devices; b) checking (5102; 5202) whether the received message is a motion message, and if so, sending (5103; 5203) the motion message; c) checking (5104; 5204) whether the received message is a button-pressed message, and if so, sending (5105; 5205) a first motion sequence; d) checking (5106; 5206) whether the received message is a button released message, and if so, sending (5107; 5207) a second motion sequence.
[13]
A computer-implemented method (5100; 5200a; 5200b) according to claim 12, further comprising the steps of: e) checking (5108; 5208) whether the received message is a scroll-up message, and if so, the sending (5109; 5209) a third motion sequence; f) checking (5110; 5210) whether the received message is a scroll-down message, and if so, sending (5111; 5211) a fourth motion sequence.
[14]
A computer-implemented method (52004; 5200b) according to claim 12 or 13, wherein step a) further comprises determining (5201) from which pointing device the message originated and determining a time of receipt of that message; and the method further comprising a step of checking (5212) if only one pointing device was used for at least a predetermined period (AT), and if so, sending the message without conversion to a motion sequence.
[15]
A computer program product containing executable instructions that perform a method according to any one of claims 12 to 14, when executed by a computer device (4501; 4601) to which a first and second pointing device are connected, and which includes a graphical user interface that includes a native mouse cursor provided.
[16]
A pointing device (5303L, 5303R) comprising: - a motion detecting mechanism (2321) for detecting a two-dimensional motion; - at least one button (M1B1, M1B2) can be pressed and released by a user; - a button detecting mechanism (2322) for detecting whether the at least one button is pressed or released; - a control unit (2324) connected to the motion detecting mechanism (2321), and provided for sending a motion message (mov2) when motion is detected; characterized in that - the control unit (2324) is further adapted to transmit a first motion sequence when it is detected that the at least one button is pressed; and for transmitting a second motion sequence when the at least one button is detected to be released.
[17]
A pointing device (5303L, 5303R) according to claim 16, further comprising at least one scroll wheel rotatable by a user; and - a scroll wheel detecting mechanism (2323) to detect whether the at least one scroll wheel has rotated; and - wherein the controller (2324) is connected to the scroll wheel detecting mechanism (2323); characterized in that the controller (2324) is further configured to transmit a third motion sequence when it is detected that the scroll wheel is being turned up; and wherein the controller is further configured to transmit a fourth motion sequence when it is detected that the scroll wheel is being turned down.
[18]
A hub (5420) comprising: - a first interface (Fig. 27: CON1; RFX1) for communicating with a first pointing device (M1), and - a second interface (CON2; RFX2) for communicating with a second pointing device (M2), and - a third interface (CON3) for communicating with a computer device (2701); - a control unit (2744; 2745) connected to the first interface for receiving first messages from the first pointing device (M1), and connected to the second interface for receiving second messages from the second pointing device (M2), and connected to the third interface for sending messages to the computer device (2701); characterized in that the control unit (2744; 2745) is adapted to perform a method according to any one of claims 12 to 14.
[19]
A computing device (4501; 4601; 5301; 5401), comprising: - an operating system (4560; 4660; 5360; 5460) with a graphical user interface that provides a native mouse cursor (705); and - a computer program product (4531; 4631; 5351; 5451) according to claim 11.
[20]
A computing device (4501; 4601; 5300; 5400) according to claim 19, further comprising: - a computer program product (4563; 4663) according to claim 15.
[21]
A computer system (4500; 4600; 5300; 5400), comprising: - a computer device (4501; 4601; 5301; 5401) according to claim 19 or 20; - a first and a second pointing device operably connected to the computer device.
[22]
A computer system (5300) according to claim 21, wherein the first and second pointing device are pointing devices (5303R, 5303L) according to claim 16 or 17.
[23]
A computer system (5400) according to claim 21, further comprising a hub (5420) according to claim 18 operably connected between the computer device (5400) and the first and second pointing devices (5403R; 5403L).

类似技术:

---

公开号 | 公开日 | 专利标题

---

US9465457B2|2016-10-11|Multi-touch interface gestures for keyboard and/or mouse inputs

---

US10133396B2|2018-11-20|Virtual input device using second touch-enabled display

---

US9389777B2|2016-07-12|Gestures for manipulating tables, charts, and graphs

---

JP3431795B2|2003-07-28|Window resizing method and computer system

---

US11068149B2|2021-07-20|Indirect user interaction with desktop using touch-sensitive control surface

---

US20120092253A1|2012-04-19|Computer Input and Output Peripheral Device

---

US20040150664A1|2004-08-05|System and method for accessing remote screen content

---

JP2002099370A|2002-04-05|Method and system for switching virtual desktops and computer program product

---

BE1026980B1|2020-08-24|WORKING METHOD AND EQUIPMENT AND SYSTEM WITH DOUBLE MOUSE SUPPORT

---

NL2022858B1|2020-10-08|WORKING METHOD AND EQUIPMENT AND SYSTEM WITH DOUBLE MOUSE SUPPORT

---

NL2022859B1|2020-10-08|WORKING METHOD AND EQUIPMENT AND SYSTEM WITH DOUBLE MOUSE SUPPORT

---

NL2021697B1|2019-04-08|METHOD AND DEVICE AND SYSTEM FOR PROVIDING DOUBLE MOUSE SUPPORT

---

US20130205201A1|2013-08-08|Touch Control Presentation System and the Method thereof

---

WO2019063496A1|2019-04-04|Method and device and system for providing dual mouse support

---

EP3683659A1|2020-07-22|Method and device and system with dual mouse support

---

US20200233504A1|2020-07-23|Method and device and system for providing dual mouse support

---

Baudisch et al.2002|Drag-and-Pop: Accelerating Drag-and-Drop on Large Screens

---

Yang2013|Blurring the boundary between direct & indirect mixed mode input environments

---

Denter2010|PyMT: New and Advanced Interaction Widgets

同族专利:

---

公开号 | 公开日

---

BE1026980A1|2020-08-17|

---

US20200233501A1|2020-07-23|

---

BE1026977A1|2020-08-17|

---

BE1027027B1|2020-09-02|

---

US11016588B2|2021-05-25|

---

BE1026977B1|2020-08-24|

---

BE1027027A1|2020-08-28|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

EP0538745A2|1991-10-21|1993-04-28|International Business Machines Corporation|Method for buffering high bandwidth data from an input device|

---

EP0635780A1|1993-07-21|1995-01-25|Xerox Corporation|User interface having clicktrough tools that can be composed with other tools|

---

US5666499A|1995-08-04|1997-09-09|Silicon Graphics, Inc.|Clickaround tool-based graphical interface with two cursors|

---

US5805163A|1996-04-22|1998-09-08|Ncr Corporation|Darkened transparent window overlapping an opaque window|

---

US6509889B2|1998-12-03|2003-01-21|International Business Machines Corporation|Method and apparatus for enabling the adaptation of the input parameters for a computer system pointing device|

---

US7802265B2|2004-03-15|2010-09-21|Imi Innovations, Inc.|Computer interface system using multiple independent graphical data input devices|

---

US8793620B2|2011-04-21|2014-07-29|Sony Computer Entertainment Inc.|Gaze-assisted computer interface|

---

US9483155B2|2012-09-13|2016-11-01|Microsoft Technology Licensing, Llc|Pointer unification|

---

US10168775B2|2012-10-10|2019-01-01|Innovative Devices Inc.|Wearable motion sensing computing interface|

---

TW201435720A|2013-03-01|2014-09-16|Hon Hai Prec Ind Co Ltd|Cursor of mouse control method|

---

TWI489324B|2013-07-05|2015-06-21|Pixart Imaging Inc|Pointer positioning method of pointer device|

---

JP5939581B2|2013-07-08|2016-06-22|インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ|Method for moving object to be moved displayed on display screen, electronic device and electronic device program thereof|

---

WO2019063496A1|2017-09-29|2019-04-04|Inventrans Bvba|Method and device and system for providing dual mouse support|

---

法律状态:
2020-10-12| FG| Patent granted|Effective date: 20200824 |

优先权:

---

申请号 | 申请日 | 专利标题

---

BE20195035A|BE1026977B1|2019-01-20|2019-01-20|METHOD AND DEVICE AND SYSTEM FOR PROVIDING DOUBLE MOUSE SUPPORT|

---